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Overview 


Introduction 


Young children and their families benefit from early care and education (ECE) settings implement ing high- 
quality practices that support children’s well-being, health, and safety. Activities to support quality are a 
focal point of federal and state investments, including over $1 billion of the Child Care and Development 
Fund and nearly $240 million in Head Start Training and Technical Assistance.? Quality improvement 
activities include a variety of delivery modes and content areas to address children’s developmental needs 
and to align with the knowledge and skills of the ECE workforce. Quality improvement in ECE is challenging, 
and sustained changes in quality across ECE settings have not been widespread. 


Quality improvement methods used in health care and other fields are of interest to ECE because of the 
research base indicating their effectiveness and because of their focus on both the content and the process 
of improvement. The Culture of Continuous Learning (CCL) project tested the Breakthrough Series 
Collaborative (BSC), a quality improvement methodology developed by the Institute for Healthcare 
Improvement. As part of the CCL project, ECE centers participating in the BSC learned new strategies for 
improving evidence-based practices. Participating centers focused on improving teaching practices 
associated with children’s social and emotional learning (SEL). Teams worked within their centers and across 
other centers participating in the project as part of a learning collaborative ona set of shared goals for 
change. They received support from experts and reflected on and shared their learning across centers. A key 
difference between the BSC and other quality improvement approaches used in ECE is the focus on both 
individual and organizational dynamics that support sustained changes in practice. 


Purpose 


The purpose of this report is to share findings from the BSC feasibility study in ECE centers. The feasibility 
study examined how ECE centers (i.e., child care centers and Head Start) participated in BSC quality 
improvement activities and the extent to which the BSC promoted changes in individual practices and 
organizational dynamics. The feasibility study assessed 1) the process of implementing a BSC in ECE centers, 
2) whether participating centers developed aculture of continuous improvement, and 3) how the BSC 
fostered sustained changes in workplace climate and individual practices to support children’s SEL. 


Findings from the CCL feasibility study can inform recommendations for implementing and evaluating aBSC ata 
larger scale by including more ECE centers of different types, and across diverse geographic areas. 


Methods 


The CCL project used acase study design and mixed methods to assess the feasibility of implementing a BSC 
in ECE centers. The Feasibility Study Team collected data at two time points from staff in the seven 
participating ECE centers and from the team that implemented the BSC. Descriptive analyses examined 
changes over time in participation in BSC activities, perceptions of the BSC, quality improvement practices, 
organizational climate, and practices to support children’s SEL. 


4 https://www.acf.hhs.gov/occ/resource/ccdf-expenditures-overview-for-fy-2018-all-appropriation-years 


https://eclkc.ohs.acf.hhs.gov/about-us/article/head-start-program-facts-fiscal-year-2019 
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Key findings 


Findings from the feasibility study provide initial evidence that the BSC is feasible to implement in ECE 
settings. However, it is anintensive quality improvement methodology that requires significant investment 
of both time and resources. ECE centers in the CCL project varied in the degree to which they fully 
participated. 


Characteristics that supported the feasibility of participating in and benefiting from a BSC included: 


e Asupportive and engaged Senior Leader who encouraged staff to take ownership of the quality 
improvement process 


e Strong organizational structures (e.g., established staff meetings, flexibility tohire additional staff or 
structure staff workdays) that could be used to support the BSC activities 


e Classroom infrastructure that facilitated collaboration between teachers (e.g., doors leading to 
adjoining classrooms) 


e = Aninitially hospitable environment for change, including high levels of psychological safety for staff in 
the program 


The CCL project documented the following initial changes across most outcomes of interest: 


e Program teams shared and adopted successful tests of practice change across teams. 


e Program teams reported new knowledge about how to make and sustain improvements using the tools 
and skills obtained through the BSC 


e Program team members became more confident in collecting and using data and recognized that 
improvement is an ongoing process 


e Teammembers gained anappreciation for seeing demonstrations of leadership across members of their 
program, regardless of a person’s job title 


e Job satisfaction changed over time among program team members in ways that aligned with their level 
of participation in the BSC 


The CCL project also documented that some ECE centers faced challenges to fully participating in the BSC 
quality improvement activities. Challenges included: 


e Staffing shortages 


e Competing demands (e.g., meetings that conflicted with BSC activities such as day-long Learning 
Sessions or hour-long calls) 


e Participation in other quality improvement initiatives or program requirements 


Future implementation of aBSC across ECE centers will need to address the supports needed for centers 
with fewer initial resources or capacity to participate fully. 


> A Senior Leader isa high-level administrator from the ECE setting (e.g., center director) who is participating in the BSC and is 
responsible for providing leadership, support, and advocacy on behalf of the BSC team. This and other terms are defined in this report’s 


glossary (see Appendix A). 
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Report Structure 


In this report, first we will provide an overview of what the Breakthrough Series Collaborative is and the 
theory of change surrounding why we expect aBSC might be a feasible methodology to close the gap 
between what ECE practitioners know and what they do, and how practice change might be spread and 
sustained in centers. Next, we will describe the extent to which centers participated in BSC activities. Then, 
we will share findings from the case study of centers that participated in the BSC-SEL. Lastly, we will address 
the implications of the findings on implementing a BSC with ECE centers in the future. 


Glossary 
Breakthrough Series Collaborative (BSC) 


e Aquality improvement methodology aimed at helping organizations build ashared knowledge of 
evidence-based best practices and anunderstanding of how to implement improvements within their 
unique contexts. 


Breakthrough Series Collaborative on Social and Emotional Learning (BSC-SEL) 


e The name of the BSC implemented in early care and education (ECE) centers focused on social and 
emotional learning (SEL). The SEL content for the BSC-SEL was based on the Social Emotional 
Foundations for Early Learning Pyramid Model. 


Culture of Continuous Learning (CCL) project 


e Aproject that included implementing the BSC methodology and studying its feasibility in ECE centers. 


Expert faculty coaches/faculty coaches 


e = Individuals with specific content expertise of the BSC who serve as advisors to the Implementation 
Team (faculty) as well as coaches to the participating BSC teams. In the BSC-SEL, faculty coaches 
included an ECE teacher, a parent, an ECE Director, an ECE policymaker, and atrainer in the Pyramid 
Model. 


Implementation Team 


e Experts in BSC methodology and ECE program operations who facilitated the BSC and oversaw 
implementation in participating centers as part of the project staff. 


Feasibility Study Team 


e Researchers who led the study assessing the feasibility of using the BSC methodology in ECE centers. 
Senior Leader 


e Ahigh-level administrator or leader from the center (e.g., center director) participating in the BSC. This 
person is responsible for providing leadership, support, and advocacy on behalf of the BSC team. They 
also focus on how successful changes tested by the team can be scaled up, spread, and sustained. 


Team Leader or Manager 


e Center staff member who oversees team activities and actively guides the work of the BSC team. They 
serve as the liaison for all BSC-related assignments and activities, including data-related activities. 
Possible Team Leaders might include an associate director, education coordinator, or family 
engagement specialist. 
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Introduction 


Young children and their families benefit from early care and education (ECE) settings that implement high- 
quality practices that support children’s well-being, health, and safety. Activities to support quality area 
focal point of federal and state investments, including over $1 billion of the Child Care and Development 
Fund and nearly $240 million in Head Start Training and Technical Assistance. Quality improvement 
activities include a variety of delivery modes and content areas to address children’s developmental needs 
and to align with the knowledge and skills of the ECE workforce. Policymakers and early childhood 
stakeholders have aninterest in promoting and investing in quality improvement strategies and 
professional development supports that facilitate positive and sustained changes in quality. Quality 
improvement efforts such as coaching and training’ have been used to varying degrees of success, with 
efforts often resulting in only temporary changes limited to an individual or classroom (Joyce & Showers, 
2002). ECE leaders and staff are seeking opportunities to test new approaches to quality improvement that 
can promote meaningful and sustained changes in practice. The purpose of this report is to share findings 
from the Culture of Continuous Learning (CCL) project, a quality improvement initiative tested in ECE 
centers aimed at supporting changes in practice by addressing individual and organizational factors that 
are associated with sustaining high-quality practices over time. 


Quality improvement methodologies used in health care and other fields hold promise for use in ECE 
because of their focus on both the content and process of change. The CCL project tested the Breakthrough 
Series Collaborative (BSC), a quality improvement methodology aimed at improving the uptake and 
sustainability of evidence-based practices through work in teams within and across organizations. The BSC 
methodology focuses on implementation in real-world settings. The methodology emphasizes spread and 
sustainability of practices through organization-wide changes in the use of data toinform improvement, 
continuous learning approaches, culture, and mindset. The BSC contrasts with traditional quality 
improvement approaches in ECE that focus more on individualized professional development. Because a 
BSC is designed to facilitate program level changes in addition to changes in individual practice, it has the 
potential to support sustainable and meaningful quality improvement in ECE centers. 


The BSC methodology was developed by the Institute for Healthcare Improvement and the Associates in 
Process Improvement in 1995 for use in the health care field. It aims to bridge the gap between what is 
known to be best practice and what is implemented in real-life contexts. ABSC typically focuses on 
supporting the implementation of improvements in one content area or domain of practice. In the health 
care field, for example, improvement activities would be focused on a particular outcome (e.g., reducing 
infections) and specific evidence-based practices associated with that outcome (e.g., hand washing). The 
BSC methodology helps organizations build ashared knowledge of evidence-based practices and an 
understanding of how to implement improvements within their unique contexts. The success of using the 
BSC methodology is well-documented in the health care and social service fields (Institute for Healthcare 
Improvement [IHI], 2003; Daily et al., 2017). However, the BSC methodology has not yet been used widely 
in the ECE field. The CCL project examined the feasibility of implementing a BSC in child care centers and 
Head Start programs and evaluated whether initial changes in practices and outcomes were achieved. The 


© See https://www.acf.hhs.gov/occ/resource/ccdf-expenditures-overview-for-fy-2018-all-appropriation-years . See also 
https://eclkc.ohs.acf.hhs.gov/about-us/article/head-start-program-facts-fiscal-year-2019. 

4 A glossary of professional development terminology defines coaching and training in the following way: “Coaching is a relationship- 
based process led by an expert with specialized and adult learning knowledge and skills, who often serves in a different prof essional 
role than the recipient(s). Coaching is designed to build capacity for specific professional dispositions, skills, and behaviors and is 
focused on goal-setting and achievement for an individual or group.” “Training is a learning experience, or series of experiences, specific 
to an area of inquiry and related set of skills or dispositions, delivered by a professional(s) with subject matter and adult learning 
knowledge and skills.” Retrieved from https://www.naeyc.org/sites/default/files/globally-shared/downloads/PDFs/our-work/public- 
policy-advocacy/glossarytraining ta.pdf. 
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BSC piloted in the CCL project focused on improving practices to support children’s social and emotional 
learning (SEL). 


The CCL project included an Implementation Team that was responsible for implementing the BSC 
methodology and a Feasibility Study Team to conduct the feasibility study. The CCL project was carried out 
by researchers at Child Trends in partnership with implementation collaborators atthe University of 
Massachusetts Boston, the Center for the Study of Social Policy, and JRA Consulting, Ltd. 


The report is structured in the following sections. First, we describe the context of quality improvement in 
ECE. We then provide further details about the BSC methodology, its theory of change and how it was 
implemented in ECE centers in the CCL project. Next, we address details about the feasibility study and the 
methods used to study implementation of the BSC in ECE centers. Findings from the feasibility study are 
presented in three sections that outline a) how ECE centers participated in the BSC, b) the extent to which 
ECE centers developed aculture of continuous learning, and c) initial evidence of outcomes of the BSC. We 
conclude with a set of lessons learned about implementation and considerations for supporting 
implementation of a BSC at alarger scale in ECE programs. 


The Context of Quality Improvement in Early Care and 
Education 


Opportunities for ECE programs to participate in meaningful quality improvement and professional 
development activities differ across states, communities, and program types; the requirements, structure, 
and delivery of quality improvement activities result in experiences for programs that vary in intensity and 
likely in effectiveness. Since each state has a unique structure to support quality improvement, either 
through their state or regional quality rating and improvement system (QRIS) and/or their state 

professional development systems, there is no single “business as usual” for quality improvement in ECE 
settings. For example, while most states have implemented a QRIS for ECE programs, the range and depth of 
quality improvement options available to programs vary. Nearly all QRIS offer coaching and consultation, 
but the number of hours, format and content varies across QRIS and across programs with different needs 
(National Center on Early Childhood Quality Assurance [NCECQA], 2020). Fewer QRIS offer peer to peer 
learning activities, though use of this format is increasing over time. QRIS typically serve child care centers 
and family child care programs; fewer Head Start programs receive quality improvement supports through a 
QRIS, even if they havea rating (NCECQA, 2020b). Head Start programs have access to supports from 
regional and national technical assistance providers, and a portion of each grant to local grantees is 
allocated for their own training and technical assistance (Harding et al., 2019). Nearly all center directors in 
Head Start report that they offer workshops sponsored by their own organization or another organization; 
most also offer coaching and consultation (Harding et al., 2019). Grantee directors are more likely to 
participate in networks of leaders and leadership institutes than center directors, indicating potential 
discrepancies in how opportunities are offered in Head Start (Harding et al., 2019). 


Despite the variation in structure and delivery of quality improvement supports, quality improvement 
activities across states, communities, and program types are typically focused on individuals’ skills and 
competencies. Coaching, consultation, and mentoring in QRIS offer strategies to help individual teachers 
and directors meet the quality standards outlined in the QRIS, including scores on observational tools such 
as the Classroom Assessment Scoring System (CLASS) and Environment Rating Scales (ERS). Less attention 
and limited technical assistance are focused on the organizational structures and workplace environments 
that can foster and sustain improvements over time. Aconcern among some QRIS leaders is that coaching 
individuals on the standards or observational tools can result in practice changes that are not sustained 
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because they are not embedded in the structures and processes of the program. They may also promote 
point-in-time demonstration of skills instead of skills that are part of everyday routines. 


A related concern among leaders of quality improvement initiatives is the difficulty of implementing 
meaningful continuous improvement practices. Though QRIS and the Head Start Training and Technical 
Assistance often aim to focus on quality improvement and continuous learning, it is challenging to put these 
priorities into practice. While two-thirds of QRIS offer technical assistance that focuses on continuous 
improvement, there is variation in how each QRIS defines and implements this focus (NCECQA, 2020a). 
Continuous improvement may include completion of an assessment and goal setting but is less likely to 
include regular data collection and improvement cycles to test changes in practices (NCECQA, 2017). The 
2016 Head Start Program Performance Standards require programs to engagein monitoring and 
continuous improvement, but evidence suggests that Head Start programs have progress to make in using 
data for program improvement (Derrick-Mills, 2015). 


Overall, participation in quality improvement occurs in the context of low average wages for center-based 
staff ($12 per hour on average®) and minimal benefits. Over half of child care teachers received public 
income supports (such as the Earned Income Tax Credit or Supplemental Nutrition Assistance Program; 
Whitebook et al., 2018). As decision-makers consider what to include ina menu of options for quality 
improvement, it will be important to address compensation, benefits, and other workplace conditions that 
make it challenging to support the well-being of early educators.‘ 


Addressing quality improvement in the current context thus requires approaches that consider the wide 
variety of experiences early educators have with quality improvement and the challenges in their 
workplaces. Approaches that acknowledge the importance of organizational structures and climate, 
collaborative approaches, and ongoing learning would be valuable in ECE systems. 


Quality improvement methods used in health care and other fields are of interest to ECE because of the 
research base indicating their effectiveness and because of their focus on both the content and the process 
of improvement. Quality improvement methods provide tools that can be used across different teaching 
practices and focus on building dynamics across program leaders and staff that can support changes in 
practice over time. 


The Breakthrough Series Collaborative as an 
Innovative Approach to Quality Improvement in Early 
Care and Education 


The BSC methodology differs from other quality improvement programs, methods, and activities typically 
used in ECE programs. A BSC focuses on organizational systems and culture by recognizing that change 
effected in one part of asystem requires changes in other parts of the system. Inthe CCL project, the BSC 
mobilized teachers, administrators, and parents, and facilitated a shared learning community to address a 
problem of practice. 


® See https://www.bls.gov/ooh/personal-care-and-service/childcare-workers.htm#tab-5. 
f The terms early educators and ECE educators are used interchangeably throughout the report to add variety for the reader. Both terms 


refer to practitioners who care for and educate young children. 


Promoting a Culture of Continuous Learning in Early Care and Education Settings 7 


The process and structure of the BSC is typically delivered over a 12- to 18-month period. In the CCL 
project, the participating teams from the ECE programs attended quarterly, day-long Learning Sessions that 
included training, technical assistance, and collaborative learning activities. Between the quarterly Learning 
Sessions, the teams participated in Action Periods during which they used Plan Do Study Act (PDSA) cycles 
to test ideas and collect data to determine how well the changes worked. Participants were expected to 
learn by doing; attitudes and behaviors were hypothesized to change during the Action Periods. Teams were 
expected to learn and build confidence that the changes they were testing on a small-scale resulted in 
improvement. The teams could test practices across other conditions and bring other colleagues into the 
tests, thereby spreading the change across the program. Overall, the teams aimed to embed changes within 
the standard practices in their programs (including professional development, hiring, policies, procedures, 
etc.) sothat changes would endure. 


The following sections provide more detail about how five strategies used in a BSC were applied in the CCL 
Project with the aim of producing and sustaining quality improvement. 


The interconnected elements of the Breakthrough Series Collaborative 
methodology 


Collaborative Change Framework based on social and emotional learning 


The Collaborative Change Framework (CCF; see Appendix D) is the content anchor for all work that is done 
by teams ina BSC. It serves as aroadmap throughout a BSC to guide ateams’ assessment of their strengths 
and challenges, opportunities for improvement, and testable changes as well progress tracking over time. 
The CCF describes what elements (primary drivers) must change and how those elements must change 
(secondary drivers) to achieve the aims of the BSC. It also includes a set of metrics that teams in a BSC use to 
identify priorities and track and assess their own progress. The CCF describes the need and opportunity for 
improving SEL, or whatever the content goals are of the particular BSC that is being implemented, in child 
care centers and Head Start and the overarching aims that the centers in the BSC seek to achieve. 


The CCF was developed by examining the research and practice guidance from the Center for the Social 
Emotional Foundations for Early Learning Pyramid Model’ and by gathering the expertise of practitioners in 
the field. In the CCL project, the areas of the CCF included: SEL practices, partnering with families, racial 
equity and cultural competency, parents, and ECE leadership. We refer to the CCF for the BSC-SEL as the 
Driver Diagram to distinguish between CCFs in BSCs generally and the CCF of the BSC-SEL specifically. The 
Pyramid Model was chosen for the project because it has been extensively studied and widely used in ECE 
programs (including early childhood special education settings). It was developed to provide ECE educators 
with guidance on how to support young children’s social and emotional development and address 
challenging behaviors. In addition, the state in which the CCL project took place had already adopted 
Pyramid Model in its ECE system. This state’s early learning standards bring attention to these critical areas 
of development and learning and give the early childhood field a framework for best practices to support the 
development of these important competencies. All licensed ECE centers in this state are expected to use 
these standards to guide curriculum and instruction. 


& The Pyramid Model is aframework of evidence-based practices that support social and emotional competence in infants and young 
children. For more information see: https://www.pyramidmodel.org/. 
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Cross-role teams 


The cross-role team, also referred to 
as the Core BSC team, is a multi-level 
team from each center participating 
ina BSC. According to the BSC 
theory of change (see Figure 1), these 
cross-role team members are 
expected to become champions who 
are essential for spreading changes 
across the center. In all BSCs, it is 
important for Senior Leaders to be 
engaged from the start so that they 
are learning with the team, helping to 
remove barriers to testing, 
communicating with the entire 
center, and identifying ways to 
sustain the changes through center's 
polices and protocols. 


In the CCL project, each participating 
ECE center formed a four- to eight- 
member Core BSC team. These 
members were expected to 
participate in the BSC activities, and 
work together to test, spread, and 
sustain changes in SEL practices at 
their center. The Core BSC team 
included a Senior Leader (typically a 
center director), a Team Leader 
(typically alead teacher or 
administrative staff member), other 
selected teachers and staff, anda 
parent representative. These multi- 
level, inclusive teams ensured that 
different perspectives were 
represented to identify opportunities 
to improve SEL practices. They 
helped develop and test creative 
solutions based on best practices and 
their own experiences as parents, 
teachers, and staff. 


Expert faculty coaches/faculty 
coaches 


Faculty coaches are critical to the 
success of the BSC methodology. 
Faculty coaches have expertise about 
the content of a BSC. They serve as 


Key BSC Terms 
Core BSC team 


The multi-level, inclusive teams from ECE centers directly participating 
in the BSC-SEL. Members included center administrators, teachers, staff, 
and parents. Members were expected to participate in BSC activities. 


Learning Sessions 


Four day-long, in-person meetings offered for all Core BSC teams that 1) 
focused on various topics, or drivers, for making improvements in SEL 
practices; 2) taught key quality improvement concepts and tools; and 3) 
provided time for teams to develop and make plans for the period of 
time in between Learning Sessions. 


Model for Improvement 


Three key questions that help Core BSC teams clarify their goals for 
improvement, establish measures to determine if a specific change 
actually leads to an improvement, and select the most important 
changes to work towards. The three questions are: 1) what are we trying 
to accomplish? 2) how do we know achangeis an improvement? and 3) 
what changes can we test that will result in improvement? 


Plan Do Study Act cycle (PDSA) 


A process that Core BSC teams used to test and make improvements 
that are focused on acommon aim. Asingle PDSA cycle involves 
planning to test asmall change, conducting the work to make the change, 
studying the effects of the change, and acting on what was learned (by 
adjusting what was done, taking a brand new approach, or scaling up the 
change). 


Action Period 


Period between Learning Sessions when Core BSC teams collected 
metrics data and tested changes in practice by using the Model for 
Improvement and PDSAs. The Sustainability Action Period took place 
after the fourth and final learning session. The focus of this final Action 
Period was on sustaining practices learned throughout the BSC process. 


Affinity group calls 


Senior Leader, Team Leader, teacher, and parent videoconferencing calls 
facilitated by expert faculty to bring Core BSC team members with 
similar roles and from different centers together to share progress, 
challenges, and ideas for PDSAs. 


Monthly metrics 


Monthly data collected by Core BSC teams and submitted to BSC 
Implementation Team to help teams quantify their progress in key 
domains of the Driver Diagram. In the CCL project, monthly metrics 
included children’s behaviors and two-way communication with families. 


BSC-SEL 


The name of the BSC implemented in ECE centers focused on SEL. The 
SEL content for the BSC-SEL was based on the Pyramid Model. 
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advisors to the Implementation Team who have expertise in the BSC methodology, and they serve as 
coaches to the participating BSC teams. 


Inthe CCL project, they represented key stakeholders in early education and had expertise in the primary 
drivers of the Driver Diagram: SEL practices, partnering with families, racial equity and cultural competency, 
parents, and ECE leadership. They played unique, multi-faceted roles in supporting the participating teams 
as well as the staff implementing the BSC. The faculty coaches helped teams apply the Driver Diagram and 
brainstorm changes totest throughout the BSC. The faculty coaches reviewed team metrics and 
implementation plans, facilitated creative brainstorming and problem-solving, shared practical tools and 
resources, and provided “real time” coaching at Learning Sessions and on conference calls. They advised the 
Implementation Team (see Appendix A for definition) in the planning of BSC activities, developing content- 
related materials and providing feedback and recommendations about the progress of the BSC teams. They 
promoted the goals, purpose, and process of the BSC, helping to build excitement, momentum, and buy-in 
from the teams. 


Shared learning environment 


The shared learning environment facilitates collaboration among all Core BSC teams across the BSC as they 
work on testing and implementing practices in support of acommon goal. The shared learning environment 
provides structure for teams to spend time together and collaborate during in-person activities, during 
phone calls, and virtually using an online discussion forum. In atypical BSC, Learning Sessions span two days, 
with phone calls scheduled between the in-person Learning Sessions. The shared learning environment for 
the BSC-SEL was fostered through a set of structured activities that included a Pre-Work Day," four in- 
person Learning Sessions, monthly conference calls that were specific to roles (i.e., affinity groups) or for the 
entire team, and an online forum discussion to support collaboration, Basecamp.' In addition to providing 
concrete guidance on SEL content and improvement tools and methods, each activity was designed to 
encourage teams to share what they were learning, including challenges, successes, and best practices. The 
shared learning environment is intended to build momentum across the BSC by supporting the team in 
working toward shared goals, celebrating successes, learning from failures, and jointly solving problems to 
address challenges. 


Model for improvement 


Core BSC teams worked on testing and implementing SEL practices at their centers using the Model for 
Improvement, which poses three simple questions: 1) what are we trying toaccomplish? 2) how do we know 
achange is an improvement? and 3) what changes can we test that will result in improvement? These three 
questions helped teams clearly define their goals and priorities for change (outlined in the CCF), use the 
metrics to understand where they can improve, know whether changes resulted in improvements, track 
progress towards the aims, gather evidence, and identify practice-based changes to test. Teams worked to 
make improvements in their centers over the course of the BSC using PDSA cycles in which teams planned 
to test a change ona smallscale; tested the change they planned; studied the effects of the change; and 
acted by making adjustments, taking anew approach, or expanding the test under different or broader 
conditions. Because of the cyclical, continuous learning approach of PDSAs, changes were first tested 


h At the Pre-Work Day, the Core BSC teams/cross role teams completea series of assignments prior to the first Learning Session. Some 
assignments are conducted during the Pre-Work Day while others are done in the time between the Pre-Work Day and Learning 
Session 1. Assignments often include developing ateam name, ateam motto, completing a self-assessment based on the Driver 
Diagram, developing a data collection plan, and creating a “storyboard” for their team. 

' Basecamp is a cloud-based application that provides tools for team collaboration online, using message boards, timelines, file sharing, 
and task assignments. Core BSC teams posted pictures of their PDSA forms, commented onothers’ posts, asked questions, and shared 
resources related to SEL and quality improvement. 
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quickly on asmall scale and then increased over time based on learning as they moved toward full, 
sustainable implementation. 


Breakthrough Series Collaborative theory of change 


The CCL Project Team developed a theory of change grounded in research and theory that hypothesizes 
how the BSC, with its five inter-connected elements, is expected to influence change and sustain 
improvements in quality (Figure 1). The aims (impact) of the BSC-SEL were toimprove SEL practices and 
classroom quality, increase children’s social and emotional competence, and reduce challenging behaviors. 
The theory of change shows the pathways of influence from the inter-connected elements of the BSC 
methodology to its outputs (the BSC activities) which activate a set of mechanisms. These mechanisms are 
the enabling factors that produce the outcomes and the intended impact. We provide a brief summary of the 
theory of change in this section. 


As participants engage with the BSC activities, they are expected to establish new organizational structures 
and processes for learning how to improve. Activities such as the Learning Sessions, the collection of 
monthly metrics, the use of PDSA cycles to test changes and monthly conference calls across teams are 
intended to support these structures. A key question for the CCL feasibility study asked about the 
feasibility of child care center and Head Start staff participating in the BSC activities. 


When teams participate in the activities and use the new structures and processes to reflect with one 
another and ask questions, they experience changes in their relationships, thinking, and mindsets. The BSC 
strategy and its outputs are hypothesized to influence relational dynamics both within and across the 
participating teams in ways that engage center leaders, teachers, and parents in collaboratively leading 
change. The BSC is also designed to shift individual mindsets from a compliance and a one-size-fits-all 
approach to aninquiry mindset focused on learning, adaptation, and testing ideas about how to implement 
improvements in practice. Improved psychological safety, team member efficacy, and relational dynamics 
contribute to a positive organizational climate. A key question for the feasibility study asked about the 
degree to which these features of a culture of continuous improvement were evident in the participating 
programs. 


The theory of change identifies short- and long-term outcomes in two primary areas: the capacity for 
improvement and the content focus of the BSC which was SEL in the CCL project. A capacity for 
improvement is reflected in outcomes such as individual and collective efficacy, organizational climate, and 
knowledge and skills about how to make and sustain improvements. Ultimate impacts for SEL include 
increased knowledge and skills related to SEL, the use of high-quality teaching practices, and strong family 
partnerships. A key question for the feasibility study asked about the extent to which participation in the 
BSC resulted in initial changes in short- and long-term outcomes. 


i A more complete description of the Theory of Changeis available in Douglass et al. (2019). 
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Figure 1.Breakthrough Series Collaborative theory of change for the Culture of Continuous Learning 
project 
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Source: Douglass, A., Halle, T., & Tout, K. (2019). The Culture of Continuous Learning Project: A Breakthrough Series Collaborative for 
Improving Child Care and Head Start Quality: Theory of Change, OPRE Report #2019-100, Washington, DC: Office of Planning, Research 
and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. 
https://www.acf.hhs.gov/sites/default/files/opre/ccl theory_of change brief.pdf 
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Studying the Feasibility of a Breakthrough Series 
Collaborative in Early Care and Education 


The purpose of the CCL feasibility study was to examine how ECE programs participated in a BSC, the 
extent to which they developed aculture of continuous learning, and whether perceived changes in 
outcomes were measured asa result of participation. The feasibility study aimed to uncover lessons about 
implementation of a BSC in ECE programs and identify considerations for future practice and research of 
the BSC quality improvement methodology in ECE programs. 


The project used a case study design to allow in-depth examination of processes and outcomes and to gain 
insights into the perceptions and experiences of participants. The approach we take throughout this report 
is to share illustrative examples using quotes and figures. Key themes represent those that are supported by 
more than one piece of evidence collected in the study. 


Site selection process 


Selection of sites to participate ina BSC is acritical activity in early implementation. The selection process 
allows prospective sites to learn about the purpose and activities in aBSC and to assess their interest and 
capacity to participate. The process alsoallows the Implementation Team to assess the applications 
submitted by prospective teams and determine their readiness for the process. This “mutual selection” 
activity is designed to promote amore successful BSC by ensuring that teams are ready to engage in the 
process and have the initial capacity to participate in the BSC activities. This implies that in theory some 
centers that apply to participate ina BSC may not be selected to participate, especially if the collaborative 
has the capacity to servea limited number of centers and the demand to participate in the collaborative is 
greater. It also informs the implementation of the BSC sothat initial planning can address the unique 
strengths and weaknesses of the teams and design appropriate supports. 


The Implementation Team distributed information and materials to email lists of ECE programs to provide 
information about the BSC.k These materials included an invitation to participate in an information session 
to learn more about the BSC. The focus of the information session was to share an overview of the 
application process and an opportunity to ask questions. 


The Implementation Team then emailed the application to prospective programs with instructions to return 
the application in one month. While programs reviewed the application, the Implementation Team 
conducted focused outreach through phone calls and emails to provide one-on-one support to applicants as 
they completed their applications by answering questions, offering guidance, and providing 
encouragement. 


Seven programs applied to participate and all seven were selected. The Implementation Team noted that 
there were very few applications from Head Start centers. Additionally, twelve centers seriously 
considered applying but ultimately did not apply. The Implementation Team completed in-depth follow-up 
interviews with directors at six of these twelve sites to learn what programs identified as major barriers to 
participation throughout the application process. Specifically, center directors reported a lack of time to 


K ECE programs that had participated in Pyramid Model training were emailed information about the project. Additional detail about 
how the Implementation Team targeted their outreach to ECE programs in the geographical area where the project took place is 
omitted because of concerns this could identify centers that participated in the project. 
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participate in the BSC, competition from other early childhood quality improvement initiatives, and 
contextual issues such as staffing shortages. 


Participants 


The CCL project included a sample of seven child care centers and Head Start programs from a large city. 
Each participating center received a stipend of $3,500 that could be used at their discretion to support their 
participation in the BSC. The centers, while all determined to be a good fit for the BSC-SEL, differed in their 
structure and staffing, ranging in size from five to almost 20 classrooms, fewer than 50 to over 200 enrolled 
children under the age of five years, and less than a dozen staff to over 30. 


With support from the Implementation Team, each of the seven participating centers formed a multi-level, 
Core BSC Team composed of representatives from their centers. Core BSC teams varied in size and makeup, 
but most were comprised of a Senior Leader! (e.g., center director), Team Leader (e.g., center administrator), 
teacher, and in some cases, a parent. The size of Core BSC teams ranged from four to eight, with six of the 
seven teams experiencing attrition in membership over time. Core BSC team members served as 
representatives for their center, participating directly in BSC activities (i.e., attending four in-person 
Learning Sessions and monthly affinity group calls, interacting on the shared online discussion forum 
(Basecamp site), collecting and submitting monthly metrics, and conducting PDSAs over a one-year period 
(see Figure 2 for adetailed timeline). Core BSC Team members from each of the centers, in addition to other 
staff members employed for more than 10 hours per week at their centers, were included in feasibility study 
data collection efforts (see Recruitment and Consent Process for more information). 


In addition to participants associated with each center, the Implementation Team and faculty coaches were 
also included in data collection efforts as key informants onthe BSC implementation. The Implementation 
Team was made up of six facilitators (including an improvement advisor trained in the BSC methodology) 
across the University of Massachusetts Boston, the Center for the Study of Social Policy, and JRA 
Consulting Ltd. Implementation Team members supported the T, meeting with them by phone over the 
course of the BSC process to provide consultation and facilitate in-person Learning Sessions. The faculty 
included five additional on-the-ground expert coaches with expertise in SEL and child development who 
supported and coached Core BSC teams about SEL and quality improvement practices. They were hired by 
the Implementation Team and received training on the BSC methodology and on their roles in the BSC. 
Faculty coaches were responsible for facilitating monthly affinity group calls with Core BSC Team members, 
leading breakout sessions and presentations at the in-person Learning Sessions and providing support to 
teams on the online discussion forum site. 


Recruitment and consent process 


The Feasibility Study Team distributed consent forms to Core BSC Team members and directors at the Pre- 
Work Day and asked for consent for the Implementation Team to share administrative data collected for 
the purposes of BSC implementation with the Feasibility Study Team. 


The Feasibility Study Team recruited Core BSC Team members and other center staff members to 
participate in direct data collection activities (i.e., surveys, focus groups, interviews, classroom 
observations). The Feasibility Study Team distributed information about the study activities to staff who 
attended the Learning Sessions to share with staff from their center who were not able to attend. All staff at 
centers who worked 10 or more hours per week were eligible to participate in the direct data collection 


' Senior leaders were usually a center director. However, some teams split senior leader responsibilities across acenter director and 
another high-level administrator or two high-level administrators. 
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activities and all eligible individuals who consented were included. To identify who was eligible, directors 
provided the names, roles, and email addresses of staff at their center who worked 10 or more hours per 
week. 


In addition to signing consent forms to participate in the feasibility study overall, participants were also 
asked for their consent at the start of each survey, interview, and focus group. The Time 1 and Time 2 
surveys distributed both online and as apaper form included astatement of consent describing the 
voluntary nature of the survey and its purpose.™ 


The study protocol and all recruitment and consent procedures and materials were reviewed and approved 
by Child Trends’ IRB and OMB (Control No. 0970-0507, Exp. 03/31/2020). 


Project timeline 


The feasibility study collected data from multiple sources across all phases of implementation of the BSC. 
Both qualitative and quantitative data were collected and used in a mixed methods analysis of evidence, 
testing the hypotheses associated with the Theory of Change and the feasibility study research questions 
(see Appendix C for a full list of measures as they align with the research questions and the Theory of 
Change). Qualitative data sources included key informant interviews and focus groups; quantitative data 
sources included participant surveys, classroom observations, and BSC artifacts and administrative data 
(e.g., Learning Session notes, monthly metrics, attendance sheets). Due to the small sample size and the 
exploratory nature of this study, statistical analyses were not conducted to detect significant differences 
between subgroups. Figure 2 shows the timeline for BSC implementation and feasibility study data 
collection. 


™ The Feasibility Study Team also conducted on-site observations using the Teaching Pyramid Observation Tool (TPOT™; (Hemmeter 
et al., 2013) and Teaching Pyramid Infant-Toddler Observation Scale (TPITOS™; Bigelow et al., 2019) in an effort to measure teachers’ 
SEL practices. The TPOT and TPITOS were chosen because they are fully aligned with the Pyramid Model. Both Core BSC teachers and 
other center staff in infant/toddler and preschool classrooms participated in on-site observations. This report does notinclude 
recruitment methodology nor findings from the observations. The TPOT and TPIT OS were designed for use in both research and 
quality improvement efforts. However, these tools are primarily for professional development purposes. While it was feasible to 
conduct on-site observations in the center classrooms that participated in the BSC, the data collected from the observations were 
difficult to interpret in the context of evaluating outcomes. 
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Figure 2. Timeline of Breakthrough Series Collaborative implementation and feasibility study activities 
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Key Findings 
The findings are reported according to the research questions emerging from the BSC theory of change. 


e Towhat extent did programs engage with the BSC and participate in the activities offered to program 
teams? 


e How successfully did programs begin developing a culture of continuous learning? 


e Was there initial evidence that suggests participation in the learning collaborative was associated with 
desired outcomes related to teaching practices and organizational culture? 


For each sub-section corresponding to these three research questions, we provide a brief overview of the 
key findings in a“findings ata glance” section and identify key data sources and measures examined for 
exploring the research question." We conclude the findings section with a discussion of the lessons we 
learned about implementation and evaluation of a BSC in ECE programs. 


Please note that, due to the small sample size of this project, some potentially identifiable information about 
the findings is purposefully not included in this report to protect the identities of the centers and center 
staff. In addition, because of the small sample size for this study, the Feasibility Study Team was not able to 
test the statistical magnitude of observed or reported changes in quality improvement or SEL practices over 
time. Consequently, any quantitative analyses presented in the findings are purely descriptive and 
exploratory in nature. 


" More detailed information on the alignment between research questions and data sources for this study can be found in Appendix B. 
Information about individual measures used in the feasibility study can be found in Appendix C. 
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Engagement and participation in the Breakthrough Series Collaborative 


Findings at aglance: Engagement and participation 


According to the BSC-SEL theory of change, a key step in ECE quality improvement is meaningful 
engagement of programs in improvement activities. 


Program participation in activities varied across programs and over time. 

e Some programs had consistent team representation at in-person activities, while others sent only 
one or two representatives or did not attend at all. 

e Participation in in-person activities decreased over time. 

e Senior leaders on teams had competing responsibilities that made it difficult for some to participate. 


e Indicators of program participation were compiled to identify two groups of programs: two 
programs with “robust” participation and five programs with “moderate” participation. These 
groups of participants differed on other organizational characteristics, with robust participation 
programs appearing to have greater capacity for engagement at the outset of the project. 


Programs completed datacollection onthe monthly metrics. 


e Allprograms were able to collect and submit monthly metrics data, but this process was more 
challenging for someto complete on aregular basis. 


e Identifying astaff person to lead the metrics collection facilitated the process. 


Participants enjoyed the learning collaborative and found it to be different than other quality 
improvement initiatives. 


e Participants reported positive perceptions of the activities in the learning collaborative. 


e Participants valued the in-person meetings and the opportunities to interact with other team 
members through monthly calls. 


e Participants liked the team and group coaching they received on SEL practices. 


To understand engagement and participation in BSC activities, we examined artifacts of the BSC process, 
such as attendance lists for Learning Sessions and affinity group calls, and the number of monthly metrics 
centers gathered and shared during Action Periods. Additional artifacts of the BSC process included 
analyses of online discussion forum posts and the “yearbook” assembled for each center by the 
Implementation Team during the Sustainability Action Period that summarized in the form of afew 
PowerPoint slides what the center did while participating in the BSC. Qualitative datasources included 
interviews with Implementation Team members, interviews with faculty members, interviews with center 
directors, and focus groups with Core BSC teams. Together, these data offered a portrait of how teams and 
individuals participated in the BSC. 


Participation in the Breakthrough Series Collaborative varied across and within centers 


The Feasibility Study Team examined the level of centers’ participation, engagement, and enthusiasm for 
Learning Sessions, monthly metrics, calls (e.g., affinity group calls, BSC team calls), site visits, and the online 
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discussion forum. Figure 3 demonstrates that Core BSC teams’ members participated in the in-person Pre- 
Work Day and Learning Sessions to varying degrees. While centers were informed of and aware of the 
expectation to participate in Learning Sessions when they applied for the project, participation at in-person 
sessions varied over time and across sites, despite the logistical support that center directors reported they 
did to support staff attending in-person activities, such as organizing transportation, creating opportunities 
for collaboration asa Core BSC Team, and encouraging enthusiasm for the BSC-SEL. Core BSC teams 
fluctuated in size as teachers sometimes stepped in to take the place of another who could not participate, 
but the range over the course of the BSC was 3-8 members per center. Variation in Learning Session 
participation occurred within centers over time, and alsoacross centers. For example, a Team Leader from 
Centers B, D, and G attended each of the five all-day, in-person convenings. And a Team Leader from 
Centers A and E attended four of the five all-day, in-person convenings. A Team Leader from Center C did 
not attend any of the in-person meetings, although a Senior Leader did attend at least two of the five in- 
person convenings. 


Figure 3.Core Breakthrough Series Collaborative Team members attendance at in-person activities 
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Within a Core BSC Team, the Senior Leader is expected to play an important role in enabling participation, 
facilitating changes in the programs to support improvement, and helping set the tone and model to others 
that participating in BSC activities is valuable. As seen in Figure 4, Senior Leaders varied in their attendance 
on conference calls. Senior Leaders did not all consistently participate in the Senior Leader Call, with only 
three Senior Leaders (in Centers A, C, and D) attending more than half (between 60% and 90%) of the calls. 
Senior Leaders in four of the seven centers attended between one and three Senior Leader Calls out of the 
10 that were held. 
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Figure 4. Senior Leader calls attended by center 
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Source: Implementation Team records. 


During the in-depth phone interviews the Feasibility Study Team conducted with Senior Leaders, they 
expressed their interest in participating in calls, and their perception that attendance was beneficial. 
However, other obligations such as impromptu meetings sometimes prevented them from more fully 
participating in all activities, including the Senior Leader calls. 


It [the Senior Leader activities] was helpful if | could have participated more. | think that my position 
here and also having to attend meetings or impromptu meetings sometimes...those prevented me 
from participating inthe BSC as | wanted to asa... leader. - Senior Leader 


| like the conference calls personally because it makes me feel important. Because your voice is 
being heard...You'd be surprised when you hear different centers going through the same thing 
you're doing. And most of the time, you're really helping someone out when they hear your voice, 
and when you hear their voice, they're helping you out. - Senior Leader 


Documentation of monthly metrics varied by center 
Monthly metrics are the data that Core BSC teams collect and submit tolmplementation Team Members 


about children’s behaviors and two-way communication with families. Documenting and measuring metrics 
over time is away to quantify progress Core BSC teams make in key domains of SEL over the course of the 
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BSC. The BSC lasted 12 months, so if centers had collected data every month, they would have data for 12 
months on each of these two topics. 


The number of monthly data points collected by each center varied, as seen in Figure 5, in large part due to 
not having enough staff and/or due to not having an existing data system in place. Three centers collected 
some dataon children’s challenging behaviors for seven or more consecutive months (Centers A, D, and G). 
Two centers (Centers Aand D) collected data on challenging behaviors consistently (i.e., for 10 or more 
months), while two others (Centers E and F) only submitted metrics on challenging behaviors for one month. 


Figure 5. Number of months centers submitted metrics on challenging behaviors 
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As seen in Figure 6, three centers (Centers A, D, and G) collected some data for six or more consecutive 
months about communication between teachers and families. Center B collected metrics on two-way 
communication between teachers and families for two months, while Center F collected these metrics for 
one month. 
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Figure 6. Number of months centers submitted metrics on two-way teacher-family communication 
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Overall, four teams struggled to collect monthly metrics on a regular basis. There were three centers 
(Centers A, D, and G) that collected data on challenging behaviors and two-way communication relatively 
consistently. 


While other monthly metrics were developed by the Implementation Team based on the Change 
Framework, Core BSC teams were only asked to report on these two monthly metrics. It is not unusual for 
BSC teams to collect dataon asubset of the metrics. Implementation Teams tailor each BSC to the capacity 
of the teams that participate. The centers that participated in the BSC did not have robust existing data 
systems and spent time at the beginning of the project learning and working on establishing a process for 
collecting data. To support teams in collecting data, the Implementation Team encouraged them to focus on 
the two metrics that were most important and essential to their work. This allowed teams the space to focus 
on process (tracking metrics) and some content (SEL practices). 


The qualitative data provide further insight about the variation seen across centers in their ability to collect 
the monthly metrics. For example, one director reported being short on staff, which made it hard to meet 
this expectation. 


[Discussing collecting monthly metrics] collecting the data from teachers, we have specific times, 
one day a week to collect. But then, on those days, it might not be the right time to be able to do the 
observation and provide the information for meto enter. So that is where it was difficult. We had to 
figure out to switch it - if it's not Tuesday, switch it on a Thursday, whether it's morning, we started 
switching it in the afternoon to see what works well. Again, with staffing, sometimes the teacher 
doing the data is out. - Core BSC Team Member 
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Another director indicated their center hired a special project person who was tasked with collecting the 
monthly metrics. This center had more consistently collected and submitted monthly metrics over the 
course of the project. Not all centers may have the resources to hire a special project person to manage the 
monthly metric data collection. 


..using the observations and the collection [monthly metrics]...That's where the special project 
person was really good. He was good around taking the data, analyzing it. Because if they can't see 
the analysis of something, it makes it hard from them to understand why do [sic] we keep doing this. 
- Senior Leader 


Participants felt the Breakthrough Series Collaborative on Social and Emotional 
Learning was different than other professional 


development experiences 
“When we're doing the BSC, it's a team 


Participants discussed the BSC in comparison to other “trainings.” approach. So, we have management, we 
Trainings, workshops, or classes were the typical frame of have the teachers, the supervisor, you 
reference for the ECE staff, and they talked about the activities in know, all looking at data...And of course, 
these terms. For example, they frequently referred to the in- there are those [Learning Sessions], 
person Learning Sessions as a “workshop.” This is important everyone is coming together. So, they are 
because participants were comparing the BSC to “business-as- learning asa group, similar to if they went 
usual” professional development methods. out and did an intentional training...” - 


Senior Leader 
The Feasibility Study Team interviewed Senior Leaders to better 


understand their experiences with the BSC and their perceptions about how it compared to other ECE 
trainings. Two Senior Leaders commented that the on-going support their centers received from the 
Implementation Team made it more hands-on, and therefore they were more easily able to apply what they 
were learning in their center. This was in contrast to their experiences in other professional development 
and quality improvement activities. And, a Core BSC team member reflected how they felt closer to their co- 
workers as aresult of participating in the BSC. 


One of the things | liked about this program was that it would be more hands-on rather than just 
going to a training and then after the training's over, you try to implement the program...they assist 
us with implementation which was one of our issues from the very beginning. - Senior Leader 


Other models that we’ve taken... if we go to class individually or even if we did it together, there 
wasn't a lot of follow-through. What | like about the BSC is the follow-through tohelp you 
implement what you’re learning. - Senior Leader 


It was always a teamwork type of aura. It was never a part where you're singled out. Coming 
through the door where you're just one person. | come in now, it's a group of four, and we're all in 
the same field, so we're all like one small family. That was really helpful and really good for me. 

- Core BSC Team Member 
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Participants had very positive perceptions of the Breakthrough Series Collaborative on 
Social and Emotional Learning 


Core BSC Team members participated in a focus group with their team to describe their perceptions and 
experiences with the BSC.° Members reported that the Learning Sessions were engaging and inspired them 
when they returned to their centers. One Core BSC Team member said, “When we did our first [Learning 
Session] - when we came back, we were just excited to get it going.” At another center one member said, “It 
[the BSC] was a lot of fun. All the games were so fun. Tohave awhole group running and laughing [at 
Learning Sessions] ...” A Core BSC Team member at the same center said, “What | liked was switching it up 
and talk to people and give you thought-provoking questions. | learned a lot about other people’s culture.” 


The Feasibility Study Team also interviewed six of the seven directors individually during no less than two of 
the three interview time points and they were asked to reflect on the BSC-SEL. One director pointed out 
that the Learning Sessions were unique because it gave the staff anopportunity to be with staff from other 
centers and it allowed them to focus on the content of the day: 


They [Core BSC Team members] were excited about, they liked the trainings [Learning Sessions] 
because they are away from everything. They don’t get the opportunity all the time. They are away 
from everything and they can actually concentrate. It was actually good for them to be around other 
programs. 


Twogroups of centers appeared to emerge basedon their participation profiles 


The theory of change posits that participation/engagement in BSC activities (as documented in the outputs 
of the BSC-SEL) activates mechanisms that lead to sustained changes in practices at the individual and 
organizational levels. The Feasibility Study Team examined how participation varied across centers. Some 
BSC activities were weighted more heavily than others in the Feasibility Study Team’s analysis of level of 
participation. In particular, attendance at Learning Sessions and on Senior Leader calls were weighted the 
most, while participation on Basecamp was weighted the least (see Table 1). |In-person activities were given 
the most weight because they were the primary way in which centers received information about SEL 
practices and quality improvement strategies. They prepared centers for all the work they would go on to do 
during actions periods, including participating in phone calls, conducting PDSA cycles, and collecting and 
submitting monthly metrics. Senior Leader calls were particularly important and weighted more heavily 
because the Senior Leader was responsible for supporting participation in all other aspects of the BSC (e.g., 
calls, metrics, PDSAs, etc.). They created the space to support their center’s participation and are key in 
facilitating buy-in and supporting spread in practices and knowledge sharing across their center. The 
Feasibility Study Team asked the Implementation Team to review the proposed weighting of the BSC 
activities for the purpose of categorizing level of participation; there was agreement across the Feasibility 
Study Team and the Implementation Team in the proposed differential weighting of BSC activities. 


Based on the number of activities (outputs) in which centers participated and the differential weighting of 
those activities as described earlier, two groups appeared to emerge.’ Two centers whose weighted 
participation scores were above the third quartile of scores, suggesting that they participated in BSC 
activities more frequently and/or in the activities that were believed to be most influential in supporting 
quality improvement, were categorized as having ‘Robust’ participation while the five centers whose 


° Five of the seven centers participated in the teamfocus groups. To protect the identities of the centers, we do not include additional 
details about which centers participated in the focus groups. 

P Please recall that due to small sample sizes, we could not test for statistically significant differences in level of participation. These 
designations were based on observed differences in weighted participation scores. 
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weighted participation scores were below the third quartile, suggesting that they participated in BSC 
activities less frequently and/or in activities that were believed to be less influential in supporting 
improvements, were categorized as having ‘Moderate’ participation. The categorization of centers into 
these two participation groups was used as astructure for examining subsequent process and outcome 
measures associated with the remaining two research questions. However, due to the small sample size, 
statistically significant differences between the groups could not be tested and apparent differences are 
only descriptive in nature. While there was variation in level of participation in the BSC activities, it should 
be noted that all seven programs that participated in this study were highly motivated to engage in an 
intensive quality improvement experience, as indicated by their self-selection to apply for the BSC-SEL. 


Table 1.Scoring metrics for how the Feasibility Study Team calculated centers’ participation in the 
Breakthrough Series Collaborative on Social and Emotional Learning 
Activity Scoring # of Instances Leechs Weighting 
O- No representatives 
present from center 
In-Person Activities 1 - One representative 
(pre-work day and present from center 
Learning Sessions) 2 - More than one 
representative present 
from center 
O- No representatives 
present from center 
1 - One representative 
All Team Calls present from center 
2 - More than one 
representative present 
from center 
O- No representatives 
present from center 
1 - One representative 
present from center 
O- No representatives 
present from center 
Teacher Calls 1 - One or more 
representative present 
from center 
O- No representatives 
present from center 
1 - One representative 
present from center 
0 - No data for the month 
from center 
1 - Center submitted at 11 months 
Monthly Metrics least one datapoint for (June 18 - 
one metric for the month April 719) 
(either challenging 
behavior or 


Senior Leader Calls 


Team Leader Calls 
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Activity Scoring # of Instances deaths Weighting 
conversations with 
families) 
2 - Center submitted at 
least one datapoint for 
both metrics for the 
month (both challenging 
behavior and 
conversations with 
families) 


O - Center submitted 
zero PDSAs in the action 
period 
1 - Center submitted one 3 Action 0.1 
PDSA in the action period Periods : 
2 - Center submitted 
more than one PDSA in 
the action period 
pfs 


O- No participation from 

center in the action 

period 

1- At least one 

interaction/post from 
Basecamp center in the action 3 Action 
Participation period Periods 

2 - At least one 

interaction post from two 

or more people from 

center in the action 

period 


Source: CCL feasibility study team. 


Implications for feasibility of participation in a Breakthrough Series Collaborative 
among early care and education settings 


An involved Senior Leader who facilitated buy-in from staff appeared to be key in supporting participation in 
the BSC because Senior Leaders had the authority to support teachers in taking off time to join calls and 
attend Learning Sessions, which were the primary ways Core BSC teams received information about SEL 
practices and quality improvement strategies. Senior Leader participation specifically may have been key in 
ensuring coordinated efforts in the center and acohesive Core BSC team. 


Infrastructure, policies, and resources to ensure space and time for the Core BSC teamto connect about 
their work during Action Periods (e.g., implementing PDSAs, collecting metrics, etc.) seemed to be key in 
supporting engagement. Without structures and resources to support datacollection and data entry, 
submitting metrics consistently may have been challenging for centers. 
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Developing a culture of continuous learning 


Findings at a glance: Developing a culture of continuous learning 


A culture of continuous learning is created when relationships and processes among teachers, directors, 
and other administrative staff in a program support curiosity, questions, and reflections about current 
practices. They are interested in learning from failures as well as successes and learning from one 
another. The BSC in the CCL project focused on sparking and amplifying these mindsets and 
relationships. 


The CCL project documented multiple indicators of aculture of continuous learning in centers. 


e Insome centers, teachers and directors entered the project with mindsets and organizational 
practices that support continuous learning. Measurement of these constructs showed little change 
over time. This orientation toward learning and new ideas may have motivated centers to apply to 
participate in the project. 


e Teachers and other staff were encouraged to take initiative and demonstrate leadership in trying 
and promoting new practices, regardless of their formal title in the program. 


e The teams interacted with other centers to share learning and identify opportunities to test new 
practices. Working with teams in other centers exposed participants to new ideas they may not 
have considered, as well as opportunities to learn from what worked well and what did not in other 
programs. 


e Teams began to experience some shifts in their mindset about how they approach working with 
children and families and how they approach relationships with co-workers. 


Some personal characteristics associated with a culture of continuous learning appeared to differ across 
the robust and moderate participation groups. (Note: Researchers did not analyze statistically 
significant differences between groups due to the small sample size.) 


e Directors’ and teachers’ beliefs in their ability to make decisions that result in positive outcomes 
(“efficacy”) are part of aculture of continuous learning. Directors and teachers in the robust 
participation group started high and stayed high on measures of efficacy. Directors and teachers in 
the moderate participation group started lower on measures of efficacy but changed positively over 
time. 


e Psychological safety refers to asense of trust and security that expression of ideas is welcome in 
the program. Measures of psychological safety did not change over time; however, directors and 
teachers in the robust participation group started and stayed higher on the measure than those in 
the moderate participation group. 


In this section, we present findings associated with how change was activated among participants and in 
centers participating in the CCL project. The findings suggest that by engaging in the BSC activities, 
participants may have experienced shifts in the relational dynamics within their center and in their mindset 
about how they approach their work with children and families and how they approach their relationships 
with co-workers.? These psychological changes and shifts in relational dynamics illustrate the mechanisms in 


4 Due to the small sample size, no findings were tested for statistical significance. 
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action. The Feasibility Study Team measured several constructs using standardized survey instruments, 
conducted in-depth interviews with center directors and Implementation Team members, and conducted 
focus groups with core BSC Team members. Findings from the quantitative and qualitative data collection 
are presented in this section. 


Psychological safety 


Psychological safety is an indicator of the extent to which work environments were a safe space to try new 
ideas, make mistakes, and provide input or ask for help. The Psychological Safety measure (Edmondson, 
1999) examined staff's feelings of psychological safety and learning behavior in work teams. Example items 
included, “If you make a mistake at this center, it is often held against you,” and “Teachers at this center feel 
it is safe to take a risk (e.g., trying something new in the classroom).” 


Centers characterized as having robust levels of participation (i.e., they were more involved in BSC 
activities) had staff who reported higher levels of psychological safety at Time 1 compared to centers that 
participated less frequently (see Figure 7). There was not much change in reported levels of psychological 
safety between Time 1 and Time 2, but there appear to be group differences at both time points. Centers 
characterized as having moderate levels of participation started and stayed lower in psychological safety 
over time. Due to the small sample size, no findings were tested for statistical significance. 


Figure 7. Psychological safety by participation group attime 1 and time 2 
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Source: CCL feasibility study. 
Note: Due to the small sample size, no findings were tested for statistical significance. 
Director efficacy 
Director efficacy (measures the extent to which directors feel they can make their own decisions to produce 


adesired outcome, which is another indicator of positive intra-organizational dynamics. Example items for 
directors included questions about whether directors felt respected by staff and whether they felt they 
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could effectively handle management tasks. The Feasibility Study Team used three sections from the 
directors’ sense of efficacy scale (Bloom, 2017) .The first section asks directors whether they feel that they 
have low, medium, or high levels of competence and confidence. The second section asks directors to 
indicate which of several statements best represents their perceptions of their current position with regard 
to several factors (e.g., authority, respect of staff, respect of families, use of time, management tasks). The 
third section asks directors torate their level of competence and confidence in their knowledge and skills in 
avariety of areas (e.g. promoting policies and practices that ensure children’s health and safety, building 
strong family partnerships, managing an early care and education program and its staff, and promoting 
leadership capacity at all levels of anorganization). For the director efficacy scale, the Feasibility Study 
Team calculated the scores from the three parts of the instrument to arrive at a total score ranging from 34 
to 135." As shown in Figure 8, directors from centers characterized as having robust participation in the BSC 
reported higher levels of efficacy at Time 1 compared to center directors from centers with moderate 
participation. Directors from centers characterized as having robust participation reported similar levels of 
efficacy at Time 1 and Time 2; however, center directors from moderately participating centers reported 
higher levels of efficacy at Time 2. Directors from all the participating centers reported nearly identical 
levels of efficacy at Time 2. 


Figure 8.Director/assistant director efficacy by participation group 
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Source: CCL feasibility study 
Note: Due to the small sample size, no findings were tested for statistical significance. 


Teacher efficacy 


The Teacher Sense of Efficacy Scale (Tschannen-Moran & Woolfolk Hoy, 2001; VandeWiele, 2001), adapted 
for use with ECE teachers, measures the extent to which teachers feel they can make their own decisions to 


"The descriptive scale included one question about how directors rate their levels of competence and confidence, with total possible 
points ranging from 1-3. Thesecond scale included eight questions about how the director perceived their job, with total possible 
points ranging from 8-32. The third scale included 25 questions about directors’ competence/confidence, with total possible points 
ranging from 25 - 100. The total points possible on the director efficacy scale ranged from 34 (1+8+25) to 135 (3+32+100). 
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produce adesired outcome, which is an indicator of positive intra-organizational dynamics. The teachers’ 
sense of efficacy scale consists of 20 questions. Example items for teachers included, “How much can you do 
to make positive changes in your care setting?” and “How much can you do to calm a child who is upset?” All 
teachers from the participating centers answered items about their ability to perform certain tasks in their 
current role, such as the ability to make positive changes and implement activities as planned. Respondents 
rated items ona scale of 1-9, where 1 indicated that the teacher believed their impact was “nothing” and 9 
indicated that they believed they could influence the item “a great deal.” Total scores for each respondent 
could range from 20-180, with higher scores indicating more feelings of personal efficacy for teachers. 
Figure 9 shows findings on the levels of teachers’ perceived efficacy at the start of the BSC-SEL (Time 1) and 
then at the end of the BSC-SEL (Time 2), for both Core BSC Team members and other center staff. Teachers’ 
sense of efficacy regarding their ability to perform tasks and make positive changes increased over these 
two timepoints, with a change in mean across all teachers from 133 to145. 


Figure 9. Teacher efficacy by participation group 
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Source: CCL feasibility study. 
Note: Due to the small sample size, no findings were tested for statistical significance. 


Centers with moderate participation in the BSC-SEL appear to have had more room to grow in their efficacy, 
whereas the staff at centers with robust participation seem to have started with higher levels of self- 
efficacy, and they remain high (but teachers’ efficacy drops at Time 2). There appears to be convergence in 
efficacy scores at Time 2 across the two groups of centers. 


Inter-organizational learning 


Inter-organizational dynamics capture the relationships and interactions among participating centers. The 
interviews included questions about perceptions of the shared learning environment (e.g., Learning 
Sessions, online discussion forum, calls). The responses indicated that inter-organizational dynamics shifted 
as Core BSC team members began sharing information across centers and learning from one another. The 
Feasibility Study Team asked Senior Leaders and Core BSC Team members to reflect on their experiences 
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participating in a professional community of ECE educators who were all focused on improving how they 
support children’s SEL. The responses indicated that senior-level staff and staff just entering the field valued 
being able to collaborate with others. 


One center [was] trying to change their expulsion [policy]...nobody likes to kick kids out, but you 
have some kids that there’s no way to handle...you feel like you’re letting yourself down, the parents 
down. How do you deal with that...? Being able to put your minds together, get input from teachers 
and parents - that was really helpful. - Senior Leader 


As someone newly entering the field, it was nice to speak to people who have been in the field and 
have alot of experience. It was valuable for me to get their insight and to just have conversations...it 
gave me insight into what kind of mindset do | need toapproach the work with. - Core BSC Team 
Member 


The team learning climate was also captured using Nembhard’s inter-organizational learning activity items 
(Nembhard, 2012). The intent was to capture the team learning climate among teams. Inter-organizational 
items askedrespondents to indicate the extent to which their teams interacted with others at Learning 
Sessions and on the online discussion forum (i.e., Basecamp), used monthly metrics, and joined affinity group 
calls. 


In the Time 2 Survey only, respondents were asked to self-report on their participation in BSC activities 
(e.g., attending a Learning Session, doing a PDSA). Respondents could answer with one of the following 
choices: “I don’t know,” “never,” “rarely,” “sometimes,” “regularly,” or “alot.” Only Core BSC Team members’ 
responses were analyzed to assess inter-organizational dynamics (Nembhard, 2016). Core BSC Team 
members were most likely to report actively engaging in interactions during Learning Sessions; almost all 
(94%) of the 16 Core BSC team member respondents reported that they engaged in interactions during 
Learning Sessions “regularly” or “a lot.” Core BSC Team members reported being less involved in affinity 
group calls, reporting monthly metrics, and interacting on Basecamp; the plurality of respondents said they 
“sometimes” engaged in these three BSC activities (see Table 2). 


nu DG 


Table 2. Summary of Core Breakthrough Series Collaborative Team members’ responses to self-report 
of engagement in Breakthrough Series Collaborative activities 


| don't know| Never | Rarely | Sometimes | Regularly | A lot 


Interactions during 
Learning Sessions 6% 0% 0% % 31% 63% 
(n = 16) 
M es 
onthly all-team calls 9% 0% 0% 31% 31% 31% 
(n = 16) 
1 
5 


O 3 

6 1 1 
a 
Interactions via BSC 
Basecamp 6% 0% 13% 0% 6% 25% 
(n = 16) 
Monthly metrics report 
exchange 13% 0% 7% 40% 13% 27% 
(n= 15) 


Source: CCL feasibility study, Time 2 survey. 


The Feasibility Study Team conducted in-depth interviews with the faculty members to ask about their 
reflections of how the centers were participating in the BSC. Faculty coaches have unique perspectives as 
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experts in the content (i.e., supporting children’s SEL), but new to the BSC methodology. One faculty 
member reflected about the centers’ participation: “I think people really developed a professional 
connection, for sure...It was a community that felt very comfortable and trusting in sharing what works for 
them with eachother and sharing their thoughts with each other.” 


Inquiry mindset 


Inquiry mindset captures acuriosity or willingness to try 
out new practices. People become anactive part of their 
own learning by posing questions that can be investigated 
and answered through their engagement in the learning 
process, as opposed to a one-size-fits-all approach to 
address challenging situations. In-depth interviews with 
directors and faculty members included questions about 
centers’ use of PDSAs, desire to improve practices, and 
use of data. 


“A PDSA mindset - so they are approaching 
things asa PDSA. Whether [they were] tracking 
that. | didn’t get the sense that [they] were 
tracking that but approaching - it changed the 
way [they] approached [their] project asa leader. 
[They were] telling other programs about [their] 
experiences with it. Soreading the community 
that way.” - Faculty member 


While centers didn’t always write down plans for PDSAs, they expressed curiosity in testing new 
approaches and reflecting on the outcomes. One director reported, 


|...take things I’ve learned through the conference calls and stuff like that back in the classroom and 
say to them [the teachers], ‘Well, did you try it this way? Let’s try it this way and see how it will 
work.’ 


Another director shared, 


| see the staff [Core BSC Team members], they take notes when things are being shared from all the 
participants [on calls]. And they say, ‘Okay, great. Let me do this in my classroom. Let me see how 
that will go in my room.’ 


Implications for feasibility of building a culture of continuous learning in early care and 
education settings 


Because centers self-selected into the BSC, some mechanisms related to improvement outcomes, such as 
positive group dynamics, trust, and a willingness to try new things, may have been activated even before the 
start of the BSC-SEL. A center that signs up to participate in anovel quality improvement program may 
already have an“inquiry mindset” or desire to try new problem-solving strategies. A BSC may be most 
feasible in ECE centers where staff are excited about the model and have a desire to try something new. 


Evidence based on the findings from the sevencenters that self-selected to participate in the BSC-SEL 
suggests that the BSC is feasible to implement in ECE centers regardless of their starting point. It is flexible 
and meets participants where they are. Evencenters that participated moderately in the BSC still saw gains 
in level of teacher and director efficacy, although levels were already high at Time 1 so gains appear modest; 
efficacy scores converged between participation groups at Time 2. Centers characterized by robust 
participation were already scoring quite high on these measures and had less room to grow, however the 
changes observed suggest that the BSC supported these centers’ strengths in ways that allowed them to 
make promising changes. 
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Initial evidence of short-term outcomes of the Breakthrough Series 
Collaborative 


Findings at aglance: Changes in teaching practices and organizational culture 


A key question when assessing the feasibility of a quality improvement method in ECE centers is 
whether it has promise to change the outcomes of interest. The CCL project targeted practices to 
support children’s SEL, knowledge, and skills related tothe process of making improvement, and 
features of workplace culture that support continuous learning. The extent to which team members in 
each program shared their new learning with non-team members and whether changes were sustained 
over the six months following participation in the learning collaborative were also desired outcomes. 


The CCL project documented initial changes across most outcomes of interest. 


e Examples of successful changes reported by participants included family engagement strategies, 
options for recognizing staff contributions, and techniques for supporting staff well-being under 
stress. Successful changes were shared and adopted across teams. 


e Center teams reported new knowledge about how to make and sustain improvements using the 
tools and skills obtained through the learning collaborative. 


e Becoming familiar with data and datacollection procedures helped program team members 
become more confident in their work and recognize that improvement is an ongoing process. 


e Teammembers gained anappreciation for seeing demonstrations of leadership across members of 
their program, regardless of aperson’s job title. 


e Job satisfaction changed over time among center team members in ways that aligned with their 
level of participation in the learning collaborative. Members of teams with robust participation 
appeared to have higher satisfaction scores at both time points compared to those with moderate 
participation on all the dimensions that were measured - working conditions, co-worker relations, 
the work itself and supervisor relations. 


e Some teammembers reported challenges in sharing the tools from the learning collaborative with 
other center staff. Other team members provided examples of how they shared learning across 
other staff in their centers. 


e Center team members reported little change over time on a measure of beliefs about SEL. Scores on 
the measure did not appear to differ by robust or moderate participation groups. 


Across the processes and outcomes examined, initial evidence indicates that implementing a learning 
collaborative in ECE centers is feasible and apromising strategy for addressing challenges in current 
quality improvement initiatives. Though further testing of the methodology is necessary, the findings 
from this pilot provide decision makers with information about how challenges in ECE quality 
improvement can be addressed using a method from the health sector. 


There is initial, and promising, yet limited evidence that the centers that participated in the BSC 

experienced changes in short-term outcomes in teaching practices, organizational culture, and continuous 
quality improvement practices. The evidence is limited primarily because of the case study design and the 
small sample included in the study. We present findings demonstrating how centers made changes, spread 
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changes among their colleagues, and in a few cases, with other teams in the collaborative, and sustained 
changes six months after the BSC ended. 


To examine the short-term outcomes described in the Theory of Change, the Feasibility Study Team used 
measures to evaluate center staff's knowledge, skills, beliefs, and attitudes about SEL, family engagement, 
racial equity, cultural responsiveness and how to make and sustain improvements over time. Data were 
collected from quantitative surveys about ECE practitioners’ perceived problems, perceptions of burnout 
and emotional exhaustion, beliefs about SEL, and feelings about the work environment. We also include 
findings from in-depth interviews with Implementation Team members and interviews with center directors 
as evidence of outcome data. 


In this section, we will present evidence about how the BSC supported centers in applying skills in SEL and 
improving their organizational climate and culture, including stronger leadership across organizational 
levels, and increased sense of efficacy.’ We also present how centers sustained changes in quality 
improvement during and after the BSC, including their experiences using data and their perceptions about 
the helpfulness of BSC activities and processes. 


No changes in beliefs about social and emotional learning 


To examine how center staffs’ beliefs about SEL changed during the BSC, participants completed survey 
questions at Time 1 and Time 2. Survey items asked respondents to rate their level of agreement with 
statements such as “Taking care of children’s social and emotional needs comes naturally to me,” and “The 
culture in my center supports the development of children’s social and emotional skills.” There were 
virtually no differences in the endorsement of positive beliefs about SEL by the two participation groups 
(see Figure 10). SEL belief scores stayed mostly consistent from the beginning to the end of the BSC, with 
virtually no difference between groups based on level of participation in the BSC structures and processes. 
It is important to remember that centers that participated in the BSC were required to have completed 
trainings on SEL before applying to join the project. It may be that more sensitive measures are needed to 
capture changes in practice beyond those identified in the beliefs about SEL measure. 


5’ The CCL projectused a tool capturing teacher (Tschannen-Moran & Woolfolk Hoy, 2001; VandeWiele, 2001) and director efficacy 
(Bloom, 2019) to measure how centers were developing a culture of continuous learning. The findings from those tools were reported 
earlier in this report and are not repeated here. The CCL project acknowledges that the BSC theory of change represents cycles of 
improvement such that there is an expectation of improvement over time in the same elements (e.g., efficacy) over time. 
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Figure 10. Social and Emotional Beliefs Scale by Participation Group 
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Source: CCL feasibility study. 
Note: Due to the small sample size, no findings were tested for statistical significance. 


Changes in organizational climate were associated with participation levels 


To examine changes in organizational climate, center staff reported on how satisfied they were at their job 
using the Early Childhood Job Satisfaction Survey (ECJSS; Bloom, 2010). The first part of the ECJSS asks 
respondents to indicate how strongly they agreed or disagreed to severalstatements about different 
aspects of their work life. The 50 questions in Part | of the ECJSS were organized into the following 
categories: co-worker relations, supervisor relations, the work itself, working conditions, and pay and 
promotion opportunities. For each category of Part | in the ECJSS measure, there was stability or an 
increase in job satisfaction from Time 1 to Time 2 over the course of the BSC (see Figure 11). Centers 
characterized by “robust” participation in the BSC started with higher levels of job satisfaction, which 
indicates that stronger organizational climates may relate to level of participation. On dimensions where 
robust participants were already high (co-worker relations and the work itself), less growth was noted 
(perhaps as they reached a ceiling). However, growth from Time 1 to Time 2 was noted for centers 
characterized by robust participation in working conditions and supervisor relations. Centers characterized 
by “moderate” participation in the BSC were lower on job satisfaction dimensions compared to the “robust” 
participation centers. Among the “moderate” centers, growth over time was most notable from Time 1 to 
Time 2 in co-worker relations and the work itself. 


Promoting a Culture of Continuous Learning in Early Care and Education Settings 35 


Figure 11. Early Childhood Job Satisfaction Survey - Part |: Facets of job satisfaction by participation group 
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Source: CCL feasibility study. 
Note: Possible scores on the ECJSScan range from 0 - 50. 
Note: Due to the small sample size, no findings were tested for statistical significance. 


The Early Childhood Work Environment Survey (ECWES; Bloom, 2015) provides an additional portrait of 
staff's perceptions of organizational climate. Although results are only available for four of the seven 
centers due to low response rates at one of the time points,'" variation was noted across participating 
centers in the perceptions of organizational climate (see Figure 12 and Figure 13). 


Centers A and E (in purple and yellow, respectively), tended to show increases in different aspects of 
organizational climate over time. Program D (in green) showed decreases over time in 8 out of 10 indicators 
of organizational climate. Program C (in light blue) scored consistently lower than the other centers on all 
aspects of organizational climate. This program also showed less dramatic shifts over time in staff's 
perceptions of organizational climate, and had equal number of indicators (i.e., 5) show an increase over time 
as adecrease over time. It is important to note that this center did not have much representation on their 
Core BSC team/cross-role team from teachers and other staff members at the center (i.e., it comprised 
mostly of senior leadership). The dotted line in Figure 12 and Figure 13 represents the national norm. 


' Centers with fewer than 5 respondents at one time point are excluded from the figure. 

“While our overall response rates for the Early Childhood Work Environment Survey (ECWES) and the Time 1 and Time 2 surveys fell 
short of our goal of 280%, it is comparable with response rates of similar studies of quality improvement collaboratives (e.g., 
Nembhard, 2012). 
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Figure 12. Early Childhood Work Environment Survey (ECWES) - Part A: Staff's ratings of organizational 
climate by center, part 1 
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Source: CCL feasibility study. 
Note: Response rates by center for pre (post): A= 11(7); C = 10(8); D=5 (10); E=8 (26) 
Note: Due to the small sample size, no findings were tested for statistical significance. 


Figure 13. Early Childhood Work Environment Survey (ECWES) - Part A: Staff’s ratings of organizational 
climate by center, part 2 
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Source: CCL feasibility study. 
Note: Response rates by center for pre (post): A= 11(7); C = 10(8); D=5 (10); E=8 (26) 
Note: Due to the small sample size, no findings were tested for statistical significance. 
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Stronger leadership across organizational levels 


“She [a Core BSC Team 
teacher] has grown quite a bit 
because she asks a lot more 
questions, she wants to have 
the lead ona lot more things, 
she wants to get alot more 
things done in terms of data 
collection and making sure 


Another short-term outcome of the BSC is improving leadership across 
all levels at an organization. The Feasibility Study Team asked BSC 
participants about their perceptions of this during group and individual 
interviews and documented examples of changes in perception about 
who took initiative and played leadership roles. For example, one Core 
BSC Team member reported during a focus group: 


| would say leadership, in our pre-K, afterschool, [infant/toddler 


classrooms has improved]- [the teacher] takes the initiative to others do the same. So, she has 
work with the children in the classrooms and share what she definitely grown and is taking a 
learned with other teachers...So adapting what they’ve learned, major step to becoming one of 
from what I’ve observed in the center, have made them better in the leaders.” - Senior Leader 


their jobs as teachers compared to what it was before. 


Increased efficacy 


Changes in self-efficacy among team members were reported across centers. For example, one focus group 
participant discussed how she felt an increased sense of individual efficacy in her practice by collecting 
monthly metrics. She reflected on how observing the metrics data helped her see where the children in her 
care needed more social and emotional support. 


Yeah, it [monthly metrics] was useful for me because | noticed then that some children had a harder 
time transitioning from one area to another during feeding or nap time. | was able to focus on that 
and change that, so they don’t cry...Writing it down and seeing it every week, | was like, ‘Oh, whoa, 
she really needs help doing this,’ so it did help me alot. - Core BSC Team member. 


Spread of quality improvement practices to other center staff 


The BSC methodology when used in other sectors has been found to accelerate the uptake of new practices 
and increase the likelihood that these practices will be spread throughout the organization and sustained 
over time (IHI, 2003; McPherson et al., 2015). There was some evidence of the spread of new practices in 
this study. For example, one center invented a phrase called, “Take 5,” to convey amessage to one another 
that they needed to take a break. This phrase became a mantra that many of the center staff ended up 
adopting as a way to decompress from a perceived stressful situation. Staff even developed a poem to share 
the policy: “Take a walk, havea talk, take a minute just to think, have acool drink, stretch your back, blow 
some steam, we know what you mean. Just TAKE 5.” This “Take 5” strategy was aligned with the fifth 
“driver” inthe CCF: Organizational Capacity and Support (see Appendix D). At the Learning Session, the 
center shared the “Take 5” motto and what it meant with the other centers from the collaborative that 
attended the Learning Session in an effort to share successful strategies to help them support children’s 
social and emotional development. The “Take 5” message was initially a PDSA cycle that turned into a 
successful strategy that spread throughout the center (see Figure 14). 
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Figure 14. Spread of the “Take 5” strategy through Plan Do Study Act cycles 
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Source: Implementation Team. 


While the qualitative data provided severalinstances of spread of practices to other center staff who had 
not directly participated in the BSC activities, the quantitative data from the study provided a more mixed 
picture. Over half of the Core BSC Team members reported using data to test small changes compared to 
14% of their peers in those same centers who didn’t participate directly in the BSC (see Figure 15). This 
suggests that it was difficult to spread specific QI strategies and tools to other staff at the center. However, 
evidence oninquiry mindsets suggested that the way the centers approached improvements at their centers 
did change over the course of the BSC even if they weren't using PDSAs formally. About a quarter of all 
respondents, both those who directly participated as well as peers in the participating centers, reported 
“sometimes” testing changes using PDSA cycles (see Figure 15). 
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Figure 15.Example 1 of spread of quality improvement practices: Use of PDSA cycles 
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Source: CCL feasibility study. 
Note: Due to the small sample size, no findings were tested for statistical significance. 


A similar pattern was found for solicitation of ideas and feedback from staff (see Figure 16). 
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Figure 16.Example 2 of spread of quality improvement practices: Solicitation of ideas and feedback from 
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Source: CCL feasibility study. 
Note: Due to the small sample size, no findings were tested for statistical significance. 


Centers expressed and shared an interestin sustaining quality improvement practices 


In sustainability interviews directors expressed their plans to 


continue to use PDSAs and data in their practice. One director “The biggest lesson for me was 
shared: "These PDSAs are very helpful...it kind of made it feel more around the data. I’ve always done 
like policy...this is how we solve problems now. There’s a problem, data, but never in that way...This 
let’s try this. And take out PDSAs and figure out the what the aS NONE broad way of gathering 
problem is.” information. Specifically, | had to 


open my mind to it too because 
originally | was like, ‘yeah, yeah, we 

already do that.’ But, we really 
weren't doing that.” - Senior Leader 


During interviews, directors and Senior Leaders talked about how 
becoming familiar with data and data collection procedures has 
helped them feel more confident about their work and how 
improvement isn’t something that is achieved and completed; 
rather, it’s an ongoing process. 


| think the challenge for a lot of early educators is the technology part of data collection. So, helping 
them with that can give them some confidence with that part and it just helps them move forward 
on that. And then we just have to be mindful that it’s an ongoing process...making sure it keeps 
moving forward. - Senior Leader 


| want to get back into using [the challenging behaviors metrics] because | think it will be very 


helpful when you sit down to talk to a parent...it is agood tool, so | want to go back to it. - Senior 
Leader 
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Successes in spread and sustainability of quality improvement practices 


During interviews and focus groups, Core BSC Team “All of our classrooms are set up where 


members discussed how they talked about what they were there is a joining door between the two 
learning and testing with their co-workers. These classrooms in the middle...What | found 
discussions appeared to inspire colleagues to try some of the was the teacher that is inthe BSC team, 
strategies that the Core BSC Team members were leading. whatever she's trying out, she would 
The qualitative interviews suggest that when teachers feel actually tell the other teacher that is nota 
safe to discuss their learnings and experiences with co- partof the BSC team aboutit, that it 
workers, they end up spreading quality improvement worked. Then the other teacher would try it 
practices beyond the Core BSC Team that attended too with a child...So it's like a ripple effect.” 
Learning Sessions and participated in phone calls and - Senior Leader 


conducted PDSAs. 


| was taking what we were learning in the infant/toddler classroom and | was bringing it here [to the 
pre-K classroom] and try to implement with pre-K. We had one child in pre-K here...he was not very 
vocal or responsive at first. | worked with teachers, and we implemented strategies we had learned 

from the BSC - now he’s one of the most talkative kids in the class, he’s come out of his shell in such 
a big way. - Teacher 


Challenges in spread and sustainability of quality improvement practices 


Despite the mechanisms in place that allow for staff to collaborate and build trust to share and spread 
quality improvement practices, teachers reported during feasibility study interviews that it was challenging 
for co-workers to experience the same understanding about the BSC methodology (e.g., PDSAs) without 
having participated in the Core BSC Team. One Senior Leader explained that “It'd be easier for the staff if 
someone different came in and did workshop on that [PDSAs] so they know hands-on how to do it.” 


“| broke it down and said, ‘well this is what | learned in this program that I’m in and this is what you 
can do.’ Some of them really spun off on it and some of them, | think, they needed to be in the Core 
team itself. Something like this, | feel like is hard because the center is so big to implement with 
everyone.” - Senior Leader 


Spread of social and emotional learning practices 


Participants reported that several new practices tested in PDSAs spread either amongst center staff or 
across the centers participating in the BSC. Most centers tested practices that centered around supporting 
children’s development and environment, teaching practices around supporting children’s social skills and 
emotional competencies, partnering with families to support children’s development, and improving their 
organizational culture. Many centers reflected they did not test many PDSAs associated with promoting 
racial equity and cultural responsiveness to support children’s development. The following examples show 
how PDSAs were spread across center staff: 


1. One center developed a practice in which teachers could excuse themselves from the classroom for up 
to five minutes to take aneeded break. A co-worker would take over for the teacher. This became 
known as “Take 5.” This practice spread across the center among staff who were not part of the Core 
BSC Team. 


2. The second was about supporting children’s social and emotional development by identifying one 
teacher to consistently be in the classroom when children were dropped off in the morning, thereby 
easing children’s transition for the day. This extended to when children went down for naps and 
transitioned to new classrooms in the center too. 
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3. The third was centers that adapted new welcoming practices at drop-off for engaging both parents and 
children that included saying to hello to parents and children by name. 


Implications for feasibility of using the Breakthrough Series Collaborative methodology 
for achieving, spreading, and sustaining desired outcomes in early care and education 
settings 


Characteristics that appear to support the feasibility of producing positive outcomes of a BSC in ECE 
included: 


1. Aninvolved Senior Leader who can build information sharing about the content and BSC strategies into 
existing systems (e.g., staff meetings, professional days) 


2. Classroom and organizational structures that facilitate collaboration between teachers 


Given that centers that participated more robustly in the BSC also had higher ratings of organizational 
climate than the moderate participation group at the start of the BSC, there may be reason to believe these 
centers had more established systems and structures to facilitate participation at the outset. 


Lessons Learned Implementing a Breakthrough Series 
Collaborative in Early Care and Education Settings 


Because the BSC methodology is new to the ECE field, it is important to understand if changes or 
accommodations are needed. In this section, we describe the adaptations made by the Implementation 
Team to accommodate leading a BSC with ECE centers. The findings in this section are themes collated from 
the qualitative data collected during the Time 1 and Time 2 interviews with the expert faculty and the 
Implementation Team. During the qualitative coding process, the Feasibility Study Team identified excerpts 
from the interviews that were about “variations or adaptations” to the BSC and how the BSC “fit or aligned” 
with the context of the ECE centers. The adaptations that the Implementation Team made in the BSC are 
summarized in this section as opportunities, challenges, and responsiveness to the local context. We conclude by 
presenting recommendations that the Implementation Team and faculty shared if they were to conduct 
another BSC with ECE centers in the future. 


The Implementation Team pointed out that thinking about “adaptations” may not be “the right word 
because [the BSC] is meant to be aresponsive model.” The findings described in this section highlight how 
the Implementation Team used the five inter-connected elements to meet teams where they started in 
order to be responsive and keep them engaged in the BSC in a meaningful way. 


Responding to opportunities 


Shortening the length of the Learning Session from two days to one day was a key change to the 
methodology. The Implementation Team knew based on their experience recruiting and selecting centers 
that ECE centers throughout the local area were struggling with hiring and retaining ECE teachers. The 
application review process, and subsequent conversations with selected teams, led the Implementation 
Team to make some changes in the initial phase of BSC implementation. Several of the selected center 
directors were concerned about scheduling their teams to be off-site for two days in a row for the first 
Learning Session. In addition, based on reviewers’ assessments of teams’ strengths and weaknesses in the 
application, the Implementation Team identified a need to provide early supports toteams to effectively 
conduct their self-assessment and form complete teams. Therefore, the Implementation Team decided to 
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modify the BSC schedule, replacing the two-day Learning Session with a Pre-Work Day and holding the first 
one-day Learning Session one month later. This enabled the Implementation Team to deliver enhanced 
supports early in the process, before the first Learning Session, to build team capacity. Shortening the length 
of the Learning Session from two days to one day meant that the seven centers only had to find substitute 
teachers or arrange for Core BSC Team members to be out of the classroom/center for one day, not two 
days. 


The Implementation Team launched the BSC in the summer of 2018. There were pros and cons with 
launching in the summer, especially in the ECE sector since many centers operate their centers differently 
during the school year (September - May) compared tothe summer months (June - August). One advantage 
of launching the BSC in the summer allowed participating centers to havea “soft start” to the BSC. Further, 
the Implementation Team held a Pre-Work Day in June 2018, a month before the first Learning Session took 
place, so the Implementation Team could fully launch the BSC during the first Learning Session. One 
Implementation Team member reported she would like to use the model of having a Pre-Work Day in other 
BSCs she leads. Completing the BSC over a 12-month timeline allowed for five in-person events (one Pre- 
Work Day and four Learning Sessions). One Implementation Team member said this “made a huge 
difference just to keep it regular.” 


The Implementation Team reflected that for the Implementation Team to have one point of contact for 
centers was essential and allowed for adeep connection with teams. In addition, this person designated as 
the point of contact already had adeep connection to the ECE field, which afforded credibility with teams. 
The point of contact sent emails to the centers reminding them about the in-person events, scheduled in- 
person center visits, and reminded the centers about the all-team calls. An Implementation Team member 
said of their Implementation Team colleague, “[their] regular connection with teams was tremendous at 
keeping this [the BSC] in the forefront.” 


In alot of the series [other BSCs | have helped lead asa member of the Implementation Team] | 
don’t think we [have someone in this role] and it often results in a little bit more distance from 
chasing teams down from trying to get people to respond to emails or calls and there’s not that 
touch point. | don’t feel like we feel as connected to the teams as we did in this one. 

- Implementation Team member 


Another implementation member said that having one person serve in the role as primary contact with 
centers... 


‘,.de-compliance-ified’ it...it was opposed to making it a project where teams were going to have to 
do something because you know someone was making you do it. It took it out of that and made it 
much more by choice. - Implementation Team member 


Another way the implementation of the BSC-SEL was responsive to centers was in the way Implementation 
Team Members described or marketed some of the processes of the BSC. They referred to the phone calls 
scheduled during the last few months of the BSC as “virtual field trips.” They noted strong attendance and 
engagement by the BSC teams. To prepare for the “virtual field trip,” Implementation Team Members asked 
teams to submit pictures of their ECE environment in advance. Teams talked about their environments in 
very engaged ways and other teams on the call responded with questions. During group calls that were 
scheduled earlier in the BSC process, Implementation Team members asked teams to come prepared to talk 
about a specific topic during the call. Using pictures as a frame for the calls later in the series made it visual 
and participatory. Framing these calls as “show and tell” resulted in more teams participating in and during 
calls. One Implementation Team member reflected that having the calls framed this way helped because 
“there was some trust stuff. | think there was feeling it was worthwhile. | think there isdevelopmental stuff 


Promoting a Culture of Continuous Learning in Early Care and Education Settings 44 


in there,” acknowledging that participation in calls atthe beginning of aseries may not be as robust because 
the trust hasn’t been built yet. 


The faculty reflected about their implementation of SEL best practices rooted in the Pyramid Model. During 
a Time 1semi-structured interview with three of the five faculty, none said they were making adaptations to 
how they support teams’ SEL practices based on the size of centers. Faculty reported they were focused on 
relationship-building with the centers and ensuring the Pyramid Model fits for individual centers and 
classrooms. 


Responding to challenges 


The Implementation Team reflected on their experience conducting BSCs in other sectors and 
acknowledged that a barrier to implementing aBSC in ECE settings is the lack of planning time that teachers 
have. They are “always in front of somebody.” The logistical challenges of squeezing in a phone call during 
one’s lunch hour is not unique to ECE; however, this BSC is unique in that it is operating in the context of 
limited ECE staff and “the fact that child care is working on ashoestring budget to begin with. There is less 
room, their margin for this group.” 


Another barrier that the Implementation Team reported as early as the Time 1 interview was about the 
level of parent involvement on each Core BSC Team. An Implementation Team member acknowledged that 
“having a parent come to an all-day meeting [Learning Session], that structure is tough...we have to think 
about another way for them to get QI [quality improvement] training.” Another Implementation Team 
member commented that “having ashared experience and bonding might help keep the parents involved.” 
One suggestion was to incorporate a parent into every PDSA and ask the parent what they think of the 
learning. Another suggestion to engage parents is to bring them together for a quarterly call led by the 
parent faculty coach. 


An Implementation Team member shared that having all-day meetings was challenging to convene during 
the day on a weekday. She reflected on the pros and cons of offering Learning Sessions on a weeknight or 
weekend. Some teams may have work policies in place (e.g., non-exempt status employees) precluding them 
from working overtime. Further, if Learning Sessions were offered on a weeknight, it would have to be done 
over the course of two evenings and one Implementation Team member worried that it would “dilute the 
process since people can only realistically do two to three hours in an evening.” 


An Implementation Team member said it is essential that BSC implementers clearly communicate in plain 
language to teams early and often about what teams will receive for attending anall-day meeting. She said in 
addition to giving astipend to teams, she recommends distributing stipends to teams the day of the meeting. 
Another element essential to creating a welcoming environment is to provide food to teams. Further, she 
highlighted that the Learning Sessions for the BSC were held ona University campus, which consists of 
many large buildings, not easily accessible by car or nearby parking. She also suggested that if a Learning 
Session were held on a weekend or evening that it would be important to offer child care: 


A stipend has to be labeled, told up front you are going to get $250 for coming for the day. And we 
have child care and we can’t give food, we couldn’t give food in the last one which | think is the 
biggest incentive. If you say there is lunch and dinner, people say wow | get a free lunch and a couple 
bucks, yeah! But you go to bring your own lunch, you got to find the place, which isn’t easy. Which 
[the site for Learning Sessions] is a nightmare to get to, poor things, and we are in another building 
which is much easier. | just think it is anunfamiliar place. - Implementation Team member 


A faculty coach reflected on their role as faculty coach and said a limitation of the BSC is that it did not allow 
faculty to interact with teachers and families: 
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That is one of the drawbacks of the BSC. By following the formula of these Learning Sessions, 
certain calls, the Basecamp, some of that really did not sit with the availability of the teachers. We 
lacked the ability to interact with families through this model. That just didn’t work. We had very 
few families participate and that should have been something of priority, especially with the 
Pyramid [Model]. - Faculty coach 


An interesting challenge that one Implementation Team member pointed out was operating a BSC in the 
context of aresearch study. The Implementation Team member commented that if the BSC were being 
administered in astate as part of astate-funded system, 


..there would be different pieces of infrastructure in place than how we were...we were more 
thinking about it in terms of the [research] study at times rather than things that you might think 
about for a collaborative or technical assistance process all the time. - Implementation Team 
member 


During atypical BSC, a research team is not part of the BSC process. For the CCL project, it was important 
for the implementation and research teams to collaborate about the timing of activities to reduce burden as 
much as possible on participants and to ensure the implementation team and the research team maintained 
aunified message whenever communicating with participants about the goals and activities within the CCL 
project. For example, members of the research team attended each Learning Session to observe and used 
the opportunity to briefly describe the purpose of the research study and the data collection activities While 
it can be challenging to implement aBSC at the same time as gathering rigorous dataon the BSC’s 
effectiveness, such coordinated activity has been successfully accomplished in other sectors such as health 
care and home visiting (Nadeem et al., 2013; Nores et al., 2018; Schouten et al., 2008; Wells et al., 2017). Itis 
necessary to establish efficacy of this quality improvement’s methodology in ECE settings, as well. We 
address some specific considerations for coordinating BSC implementation and research activities in the 
Recommendations section of this report. 


Responding to the local early care and education context 


The BSC is designed to be responsive to the context in which it is being implemented. The Implementation 
Team responded to challenges associated with staffing shortages in the ECE centers by using targeted and 
direct outreach to centers to encourage them to prioritize participation in various activities. Attending and 
participating in the BSC calls and Learning Sessions was challenging at times for centers due the shortage of 
staff and limited staff availability. It is possible that centers would have been more engaged and participated 
in BSC activities to a greater extent if the centers were adequately staffed. To encourage and spark interest 
and excitement in the BSC activities, the Implementation Team relied on human connection - phone calls - 
not email, as an outreach strategy. The Implementation Team used participation data from calls to inform 
what the Implementation Team’s next steps would be in terms of outreach. If they noticed patterns such asa 
lot of teachers missing calls, the Implementation Team project manager would call the center(s) to makea 
pitch about participating next time, to learn about what was preventing staff from joining a call, and to 
discuss what the Implementation Team could do to support centers to enable them to be on a call: 


Every week, [the Implementation Team] decided what works for our plan for the week or what we 
needed, or whatever we needed, sothen we decide how many people - what is the reasonable 
amount of people we will get on this call or that call or come to the Learning Session or whatever it 
may be....1 would make acall and find out where everybody was at. - Implementation Team member 


The Implementation Team project manager also called centers before the Learning Sessions to remind 


people to preview and describe what the Learning Session would be about so the centers could get excited 
about it ahead of time. Email was not as effective to spark centers’ interest and engagement. The 
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Implementation Team project manager described calling centers to spark interest about an upcoming 
Learning Session where centers were asked to prepare a skit for the rest of the teams. Upon calling centers 
prior tothe Learning Sessions, the Implementation Team project manager would tell them: 


And also what is going to happen during the Learning Session and for example at one Learning 
Session they all promoted what - they talked about what they did in the skit one. Or different things. 
And they were so over the moon - they are such EC [early childhood] teachers. You want the 
product and show it off and do it and people did it in all different ways and they were really excited 
that. That was what it was going to be. But when we did it on email, they weren't really sure what it 
means, and no, just going to be a couple minutes, you could play a game, and remember | was in the 
hallway, so we had been at their center, so we know. We can translate that to, ‘Oh, you’re just going 
to come and show what you did. Not anything scary or sketchy. Standing up in front of a group of 
thirty people, it is nerve wracking. So, just working with people with that. - Implementation Team 
member 


An Implementation Team member who has consulted on other BSCs reflected on leading a BSC in the ECE 
sector, which is under-resourced and under-funded and experiencing a shortage in the workforce. 
Implementing a BSC stressed under-resourced centers and this Implementation Team member reflected on 
the opportunity costs of not pushing centers harder in the beginning to participate. 


| feel in the beginning we almost backed off too much and we were being too responsive to [teams’ 
concerns about the level of effort a BSC takes]. So we backed up a little too much; | wish we hadn't 
backed up too much, but | think by doing it, it kept all the teams engaged, honestly. 

- Implementation Team member 


In summary, this Implementation Team member expressed concern that centers might have dropped out if 
they were pushed harder. The Implementation Team recalibrated their efforts based on teams’ capacity to 
engage, and by doing so, met teams where they were. 


Considerations for future Breakthrough Series Collaboratives in early care 
and education settings 


The Implementation Team and faculty discussed several considerations for future BSCs, including but not 
limited to: 


e Evaluating centers’ degree of “readiness” for and commitment to change. 
e Gaining buy-in from Senior Leaders. 
e Increasing participation by adapting to the local context. 


e Factoring in constraints to the timeline (e.g., considering starting in the summer or when the “school” 
year starts) when making decisions about when to start the BSC and when to schedule in-person 
activities and phone calls. 


e Tailoring the BSC content, in contrast to requiring it to focus on SEL. 

e Modifying the online learning community to enhance engagement. 

One Implementation Team member recommended evaluating whether centers are ready to participate in 
the change process before fully implementing a BSC. “Readiness” for change is often assessed during the 
application process for a BSC, but anongoing commitment to change should continue to be monitored 


throughout the BSC. They said that more investment needs to be put into building commitment to change, 
which means the “...whole center needs to be open minded to the changes but from the beginning with the 
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Senior Leader and the teachers being open-minded to hearing what people have to say and trying different 
things in the classroom with staff and the new models.” 


The Implementation Team member went on to say that it is key that Team Leaders and/or Senior Leaders 
are open-minded. “Their jobs are the strongest and difficult. | would just say a few people or even one 
person that is ready for new vision and can spread it like wildfire, that’s what happens.” 


When asked about other recommendations about implementing a BSC-SELin ECE settings, a faculty 
member talked about how racial equity should be more than one driver: 


| think the racial equity driver - that should be an umbrella. That shouldn’t be a driver. That should 
be an umbrella. That should be a way of thinking and taught in the pre-work session. It should not be 
thought of as adriver, it should be thought of asa shift in your thinking. And there should be 
exercise on shifting your thinking. And every Learning Session there should be a check-in to see 
how people are thinking about it. | don’t think it should be an add-on. These are the drivers, push 
SEL, do this, do that, and do things through racial equity and social justice lens and for family 
engagement. I think it should be an umbrella through it, under which all the activities happen. 

- Faculty coach 


Study Limitations 


The study was designed as an embedded case study, with an intentionally small sample size to examine the 
implementation of aninnovative quality improvement methodology. To the extent that quantitative 
measures were gathered, they are presented in this report to show patterns of findings, not to test 
statistically significant differences between centers. Further studies are needed to be able to determine the 
magnitude of links between participation ina BSC and quality improvement in early childhood 
environments. Additionally, more research with a larger sample size is needed for researchers to determine 
whether observed differences and changes among ECE centers that participate in aBSC are statistically 
significant or occur by chance. The authors do not encourage the reader todraw conclusions that are 
generalizable beyond the sevenECE centers that participated in this case study of implementing a BSC in 
ECE settings. 


The Feasibility Study Team encountered a few sampling and data limitations which are described in the 
following sections. 


Sample limitations 


The sample for this feasibility study was limited to the sevencenters that participated, and only one of those 
centers was a Head Start program, thus eliminating our ability to compare findings (albeit only descriptively) 
between child care and Head Start centers. As was noted at the time of center selection, this 
underrepresentation of Head Start centers was due to alack of applications to participate in the BSC from 
Head Start centers. 


Data limitations 


Two of the seven participating centers did not complete the Time 2 survey. One of these centers also opted 
out of all study interviews, as well as observations at Time 2. This limited the ability to look at changes over 
time in those centers and reduced the overall number of centers contributing to the data presented in this 
report for datacollected at Time 2. Two centers also opted out of sharing data with the Feasibility Study 


Promoting a Culture of Continuous Learning in Early Care and Education Settings 48 


Team, limiting possible analysis of their application materials, Basecamp postings, PDSAs, site visit notes, 
and other information collected for the purpose of the BSC. While the feasibility study did include multiple 
measures of workplace climate, the ECWES data presented in this report are based on asmaller sample of 
respondents at Time 1 compared to Time 2. ECWES data from three of the seven participating centers were 
excluded entirely because fewer than five respondents at each center completed the survey at either Time 1 
or Time 2. In addition, data collected from the onsite classroom observations were not included in this 
report. The data collected from the TPOT and TPITOS observation instruments, although developed for use 
for both training and research (Hemmeter et al., 2015), were difficult to interpret in relation to evaluating 
classroom outcomes due to our small sample size and the nature of how the measures are scored. 


Recommendations 


The BSC is asystem change methodology that engages participants across roles and levels within an 
organization to support the implementation and sustainability of evidence-based practices and continuous 
quality improvement strategies. The data collected in the feasibility study can inform future implementation 
and evaluation of a BSC in ECE settings. The following key considerations were developed based on the 
project findings and were tailored to the roles of various actors within a BSC, including BSC implementers 
(i.e., those who will be leading and facilitating a BSC), ECE practitioners and leaders, and ECE researchers. 


Considerations to support feasibility 


BSC implementers (those who will be leading and facilitating a BSC): 


e Develop recruitment tools and tailored products such as videos to show how a BSC is different from 
typical quality improvement approaches in ECE. 


e Ensure centers are “ready” to participate in the BSC by assessing whether the center has time, staff, and 
dedicated leadership available to participate in BSC activities. 


e Continue to monitor centers’ commitment to change throughout the life of the BSC. 
e Gain buy-in from Senior Leaders. 


e Beresponsive to time constraints of ECE centers as well as the local ECE context, (e.g., conducting 
shorter in-person Learning Sessions as well as briefer conference calls at times that are most convenient 
for classroom-based staff; considering the school calendar when scheduling in-person activities and 
phone calls). 


e Support teams in maintaining their core members over time. For example, to better engage parents, 
bring them together for a quarterly call led by the parent faculty coach. To better engage teaching staff 
on core BSC teams, use language that resonates with teachers such as referring to an affinity group call 
as a “virtual field trip” and “show and tell.” Lastly, if staffing challenges emerge, the implementation team 
and faculty coaches should work quickly to help strategize creative solutions to address staff-shortages. 


e Modify the online learning community to enhance engagement (e.g., prioritize a platform that allows 
teams to have supportive synchronous discussions or chats (instant messages) around testing changes 
and fostering idea sharing or inter-organizational learning instead of simply posting resources or 
posting status updates. 


ECE practitioners and center leaders: 


e Ensure centers have commitment to making improvements in the content of the BSC that is offered and 
an awareness of the quality improvement methodology and expectations for participation in the 
process. 
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e Ensure there is time for staff to participate in all-day meetings off-site, monthly phone calls during the 
day at the center, and time to think about challenges and ways to try to improve them. 


e Ensure there is at least one staff person on each team can serve as the Team Leader and primary point 
of contact to both the Implementation Team and fellow team members about participating in the BSC 
activities. 


e Ensure staff are committed to and have the capacity to collect data to inform their improvement work. 


ECE researchers: 


e Tothe extent possible, use data collected as part of the BSC process for research purposes sothat data 
collection for evaluation is more seamless for participants, and data can be analyzed and used rapidly. 


e Develop direct relationships with participating ECE centers to facilitate datacollection efforts for 
evaluation purposes. 


e Streamline datacollection tools to reduce burden even further for ECE staff participants. 


e Consider measures of change for each driver and evaluate outcomes based on team’s individual areas of 
focus. 


e Expand study efforts to more centers for alarger sample and the ability to conduct more robust 
subgroup analyses. 


Conclusions 


The work of quality improvement in ECE is challenging, and sustained changes in quality across ECE settings 
have not been widespread. The goal of the CCL project was to examine the feasibility of implementing the 
BSC, a quality improvement methodology found to be effective for bringing about practice change and 
improving outcomes in other sectors such as health care and child welfare, in ECE settings. The BSC is 
different than most quality improvement methods typically used in the ECE field because it addresses both 
the individual and organizational factors that are associated with sustaining high-quality practices over 
time. 


While the BSC is aninnovative and promising quality improvement methodology that is compatible with the 
needs and priorities of ECE centers, it does require sustained engagement and considerable time and 
resource investment. Specifically, the BSC requires a commitment of 12 to 18 months for multiple members 
of a program, including parents, to participate in the effort. Support is needed to facilitate participation, 
prevent attrition, and foster completion of expected project activities. 


The CCL project documented variation in the degree to which centers engaged with and benefitted from the 
BSC. Centers with robust participation shared key characteristics, including: 


e Strong and/or innovative organizational infrastructure 

e Supportive senior leadership 

e =High levels of initial psychological safety among staff 

While the BSC methodology is intended to “meet programs where they are” and help all programs improve 
practices and outcomes, the feasibility study findings suggest that centers with initial strengths in 


organizational climate and culture will be better able to engage with the BSC. Practitioners and/or 
policymakers considering use of the BSC with ECE centers should consider the additional supports or 
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adaptations needed to help centers with fewer initial resources or capacity to engage in this type of quality 
improvement process. 


Interviews with key informants identified that competing demands (e.g., other meetings, staffing shortages, 
parenting responsibilities) can impede engagement with the BSC methodology. Future implementation of 
the BSC in ECE settings will require proactive and ongoing attention to addressing the challenges faced by 
Senior Leaders, program staff, and most importantly parents for engaging in the BSC structures and 
processes. 


In sum, the BSC methodology testedin the CCL project has promise for ECE settings. Staff in center-based 
ECE programs participated in the activities and reported engagement in the content and the quality 
improvement process. The research team documented initial changes in practices that demonstrated 
movement toward a mindset about continuous learning and improved organizational support for 
improvement activities. The initial findings also highlight the challenges that ECE centers navigate that 
make it difficult for them to participate in quality improvement. Further research is needed on a larger and 
more diverse sample of ECE programs to determine the effectiveness of this quality improvement 
methodology within ECE programs and systems. Though further testing of the methodology is necessary, 
the findings from this pilot provide decision makers with information about how challenges in ECE quality 
improvement can be addressed using a method from the health sector that focuses on both individual and 
organizational change. 
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Appendix A. Glossary 


How the term appears in 
the middle of asentence 


Action Period Action Period 
Affinity group affinity group 


All-team calls all-team calls 


Term 


Breakthrough Series 
Collaborative (BSC or BSC or Collaborative 
“Collaborative”) 


BSC artifacts BSC artifacts 


BSC-SEL BSC-SEL 


Collaborative Change 
Framework (CCF; also 
known as the Driver 
Diagram) 


Culture of Continuous Culture of Continuous 
Learning Project (CCL) Learning Project (CCL) 


CCF 


Definition 


Period between Learning Sessions when Core BSC teams collected metrics data and 
tested changes in practice by using the Model for Improvement and PDSAs. The 
Sustainability Action Period took place after the fourth and final learning session. 
Role-specific groups (Senior Leader, Team Leader, teacher, and parent) facilitated by 
faculty coaches to bring Core BSC team members with similar roles and from different 
centers together to share progress and challenges with the BSC methodology, and to 
share ideas for PDSAs. 

Videoconferencing calls facilitated by Implementation Team Members to bring 
together all Core BSC teams to share progress, challenges, and ideas for PDSAs 
typically conducted on a monthly basis and focused on specific pre-identified topics. 

A quality improvement methodology aimed at helping organizations build ashared 
knowledge of evidence-based best practices and an understanding of how to implement 
improvements within their unique contexts. 

Administrative documents collected by the Implementation Team Members over the 
course of the BSC-SEL. These documents included ECE center applications to 
participate in the BSC-SEL, completed assignments from the Pre-Work Day, monthly 
metrics submissions, attendance trackers, PDSA trackers, site visit notes, Basecamp 
postings, and Learning Session evaluations collected from attendees. 

The name of the BSC implemented in early care and education centers focused on social 
and emotional learning 

(SEL). The SEL content for the BSC-SEL was based on the Pyramid Model. 

Basecamp is acloud-based application that provides tools for team collaboration online, 
using message boards, timelines, file sharing, and task assignments. Core BSC teams 
posted pictures of their PDSA forms, commented on others’ posts, asked questions, and 
shared resources related to SEL and quality improvement. 

A guiding document for a BSC to guide the changes tested by teams. In the BSC-SEL the 
CCF was based on the Pyramid Model that outlined key elements or primary drivers 
linked to SEL outcomes. The CCF broke down each primary driver into secondary 
drivers, or specific opportunities and strategies, that centers could try out and test over 
the course of the BSC-SEL to influence and improve children’s SEL outcomes. 

Culture of Continuous Learning project that included implementing the BSC 
methodology and studying its feasibility in ECE centers. 
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How the term appears in oe, 
: Definition 

the middle of asentence 

Multi-level, inclusive teams from eachECE center participating in the BSC-SEL. Each 
team is made up of representatives from different center leadership levels (e.g., 
program directors, administrative staff, teachers, and parents). Each team appointed a 
senior leader (typically a center director) and a team leader (typically an administrative 
staff person or teacher). These teams were direct participants in the BSC methodology. 
A guiding document for a BSC to guide the changes tested by teams. In the BSC-SEL the 
Driver Diagram was based on the Pyramid Model that outlined key elements or primary 
drivers linked to SEL outcomes. The Driver Diagram broke down each primary driver 
into secondary drivers, or specific opportunities and strategies, that centers could try 
out and test over the course of the BSC-SEL to influence and improve children’s SEL 
outcomes. 


PECESSCS~*~S ECE SSCS ary care ac education 


Embedded case study embedded case study Study design that includes embedded cross-site comparisons 
design design 


Individuals who have specific content expertise related to the BSC who serve as 
advisors to the Implementation Team (faculty) and as coaches to the participating BSC 
teams. In the BSC-SEL, faculty coaches included an ECE teacher, a parent, anECE 
Director, an ECE policymaker, and atrainer in the Pyramid model, 


Implementation Team . Sone note . 

Team/BSC Implementers oversaw implementation in participating centers as part of the project staff. 

The way people “approach their problems of practice with curiosity and a desire to test 
Inquiry mindset inquiry mindset possible solutions and strategies to learn what works and what may not work” 

(Douglass et al., 2019). 

Full-day, in-person meetings offered over the course of a BSC for all Core BSC team 

members that 1) focused on various topics, or drivers, for making improvements in 
Learning Sessions Learning Sessions practices; 2) taught key quality improvement concepts and tools; 3) provided time for 


teams to develop and make plans for the period of time in between Learning Sessions. In 
the BSC-SEL, there were four full-day Learning Sessions over a ten-month period. 


: . In the CCL Theory of Change, the dynamics and mindsets that specify how the outputs 
Mechanisms mechanisms aes . ns # 
activate changes that result in achieving the short-term and long-term outcomes. 


Core BSC teams/cross- Core BSC team/cross-role 
role team team 


Driver Diagram (also 
known as the Collaborative | Driver Diagram 
Change Framework) 


Expert faculty coaches faculty coaches 
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How the term appears in 


Definiti 
the middle of asentence een 


Three key questions that help Core BSC teams clarify their goals for improvement, 
establish measures to determine if aspecific change actually leads to an improvement, 

Model for Improvement Model for Improvement and select the most important changes to work towards. The three questions are: 1) 
what are we trying to accomplish? 2) how do we know a change is an improvement? and 
3) what changes can we test that will result in improvement? 


Monthly data collected by Core BSC teams and submitted to BSC Implementation Team 
to help teams quantify their progress in key domains of the CCF. In the BSC-SEL, 
monthly metrics included children’s behaviors and two-way communication with 
families. 


Other center staff other center staff Early care and education staff who are not members of the Core-BSC team. 


Ouiouts mutoiits Inthe CCL theory of change, the “primary activities and products of the BSC process 
P P (e.g. learning sessions, monthly metrics, Action Periods)” (Douglass et al., 2019). 


A continuous quality improvement process that Core BSC teams used to test and make 
Plan Do Study Act cycle improvements. A single PDSA cycle involves planning to test a small change, conducting 
(PDSA) PDSA the work tomake the change, studying the effects of the change, and acting on what 
was learned (by adjusting what was done, abandoning the test, or scaling up the 
change). 


Monthly metrics monthly metrics 


A series of assignments done by BSC teams prior to the first Learning Session. Some 

assignments are conducted during the Pre-Work Day while others are done in the time 
Pre-work pre-work between the Pre-Work Day and Learning Session 1. Assignments often include 

developing ateam name, ateam motto, completed a self-assessment based on the CCF, 

developing a data collection plan, and creating a “storyboard” for their team. 

An introductory in-person kick-off session held before the first Learning Session to help 
Pre-Work Day Pre-Work Day Core BSC teams get oriented to the BSC methodology, assess their center based on the 

CCF, and get to know the faculty coaches and other BSC teams. 


The feeling people develop when they “begin to feel more safety around sharing their 
Psychological safety psychological safety opinions, voicing new or different ideas, testing changes on their own, sharing data, 
learning from both successes and failures, and learning from one another.” 
The Pyramid Model for Supporting Social Emotional Competence in Infants and Young 
Children is aframework of evidence-based practices that support social and emotional 
competence in infants and young children that provided the content delivered through 
the BSC-SEL. It guided the creation of the CCF for the BSC-SEL. 


Social and emotional learning 


Pyramid Model Pyramid Model 
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How the term appears in Bac 
: Definition 

the middle of asentence 

High-level administrator or leader from the center (e.g., Center Director) participating 
in the BSC. This person is responsible for providing leadership, support, and advocacy 
on behalf of the BSC team. They also focus on how successful changes tested by the 


team can be scaled up, spread, and sustained. 

The desired increase in knowledge, skills, beliefs, and attitudes as aresult of the 
Short-term outcomes short-term outcomes ' bee aa : : ; és 

mechanisms that “activate individual and collective learning and improvement 

The secondary outputs produced by the BSC that “foster new routines for collaborative 
Structures structures : ; sir gs f ; : : Spin 

professional learning, building relationships, and improving communication. 

Center staff member who oversees team activities and actively guides the work of the 
Team leader (or team BSC team. They serve as the liaison for all BSC-related assignments and activities, 

Team Leader ‘ ‘ : aay : P ? 

manager) including data. Possible team leaders might include anassociate director, education 

coordinator, or family engagement specialist. 

The secondary outputs produced by “the BSC that teach participants how to change 
Work processes work processes practices by using tools to collect and use data to inform tests of change and 

improvement.” 


Senior Leader Senior Leader 
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Appendix B. Alignment of Culture of Continuous Learning Feasibility Study 
Research Questions and Selected Data Sources 


CCL Feasibility Study Research 
Questions 


To what extent did centers 
engage with the learning 
collaborative and participate 
in the activities offered to 
program teams? 


How successfully did centers 
did begin developing a 
culture of continuous 
learning? 


BSC-SEL Theory of 

Change Component 
Targeted 

Strategy & Outputs 


Mechanisms 


Constructs of Interest 


Learning Sessions 
Monthly Metrics 
Regular Calls (affinity 
group calls; BSC Team 
Calls) 

Site Visits 

Online Discussion Forum 
Spread of SELand Ql 
practices among and also 
beyond Core BSC Team 
members 

Psychological Safety 
Team member efficacy” 
Parallel process 
Intra-organizational 
learning 
Inter-organizational 
learning 

Inquiry mindset 


Data Sources Aligned with Constructs of Interest 


Implementation Tracker 

Interviews with Implementation Team members 
Interviews with faculty members 

Interviews with site directors 

Focus groups with Core BSC teams 

Discussion forum postings 

Key informant interviews 

BSC-SEL Implementation artifacts 


Psychological Safety (Edmondson, 1999) 
Teacher Sense of Efficacy Scale (Tschannen-Moran 
& Woolfolk Hoy, 2001; VandeWiele, 2001), 
adapted” 

Directors’ Efficacy Scale (Bloom, 2019) 
Intra-organizational dynamics scale (Nembhard, 
2012) 

Inter-organizational dynamics scale (Nembhard, 
2012) 

Interviews with Implementation Team members 
Interviews with faculty members 

Interviews with site directors 

Focus groups with Core BSC teams 


Y Team member efficacy is examined using a measure of individual efficacy that looks across teachers; There isnot a separate measure of “team efficacy.” 
“Findings from teachers within Core BSC teams will be presented and discussed in the Mechanisms section. 
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CCL Feasibility Study Research 
Questions 


Was there initial evidence 
that suggests participation in 
the learning collaborative 
was associated with desired 
outcomes related to teaching 
practices and organizational 
culture? 


What adaptations are 
needed for successfully 


implementing a BSC with ECE 


centers? 


Source: CCL Feasibility Study Team. 


BSC-SEL Theory of 
Change Component 
Targeted 

Short-term Outcomes 


Not applicable 


Constructs of Interest 


Increased knowledge, 
skills, beliefs, and 
attitudes about SEL, 
family engagement, racial 
equity and cultural 
responsiveness and how 
to make and sustain 
improvements 

Improved organizational 
climate and culture 
Stronger leadership at all 
levels of the organization 
Increased sense of 
individual and collective 
efficacy” 

Adaptation 

Responding to 
Opportunities and 
Challenges 

Responding to the Local 
Context 


Data Sources Aligned with Constructs of Interest 


Perceived Problems Questionnaire (Johnston, 
1984) (Ti vs. T2) 

Maslach Burnout Inventory: Emotional Exhaustion 
Scale (Maslach et al., 1997) (T1 vs. T2) 

Social and emotional beliefs scale (Brackett et al., 
2012), adapted (T1 vs. T2) 

Early Childhood Work Environment Survey (Bloom, 
2015) (T1vs. T2) 

Interviews with Implementation Team members 
Interviews with site directors 

Teacher Sense of Efficacy Scale (Tschannen-Moran 
& Woolfolk Hoy, 2001; VandeWiele, 2001), adapted 
(Ti vs. T2)Y 

Directors’ Efficacy Scale (Bloom, 2019) (T1 vs. T2) 


Interviews with Implementation Team members at 
Time 1 and Time 2, and sustainability interviews 

with Implementation Team members conducted in 
2020, six months after the fourth Learning Session. 


* The Teacher Sense of Efficacy Scale is a measure of individual efficacy and inthis study is being examined across teachers. There isnot a separate measure of “collective efficacy.” 
Y Findings from all center staff will be presented and discussed in the short-term outcomes section. 
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Appendix C. Feasibility Study Measures 


The feasibility study used a mixed-methods approach to capture dataon each component of the theory of 
change, from strategy through short-term outcomes.” Qualitative and quantitative data sources included 1) 
observation data, 2) survey data, 3) interview and focus group data, and 4) BSC implementation artifact 
data. Classroom observations were conducted by observers unaware of which classrooms had teachers 
participating on the Core BSC teams. 


Surveys 


Surveys included measures of organizational climate, psychological safety, and self-efficacy obtained from 
teachers and staff at participating centers both early and later in the implementation of the BSC-SEL. Both 
Core BSC Team members and non-Core BSC Team members were asked to complete surveys. 


Early Childhood Work Environment Survey (ECWES; Bloom, 2015) 


The ECWES was used to measure the organizational climate of the seven centers participating in the BSC- 
SEL. The ECWES was developed specifically for ECE settings and is administered online by an organization 
called New Horizons. All center staff at participating centers who worked more than 10 hours per week 
were eligible to complete the ECWES. The survey was fielded at two time points (i.e., early on in the BSC and 
again after the completion of the final Learning Session). The ECWES included questions about staff's 
perceptions about various organizational practices and took approximately 15 minutes to complete. Survey 
questions fell into ten dimensions (listed below) that helped the Feasibility Study Team better understand 
the collective perceptions of staff in each center. Specifically, the tool helped decipher what things were 
going well in the center and better isolated areas that need to be strengthened (addressing research 
questions 8,9, and 10). Example ECWES items about work attitude included asking the respondent to 
indicate if they “intend to work here at least two more years,” and “I sometimes feel trapped in my job.” In 
order to generate an ECWES profile, center directors also completed a Center Background Information 
Form prior to completing the ECWES. This form gathered information about the center’s funding sources, 
age groups served, and other center characteristics. After closing data collection for the ECWES, an 
electronic Work Environment Profile summarizing the aggregate results for each center was autogenerated 
by New Horizons and sent to the Feasibility Study Team to review. Results were presented on the 
dimensions below: 


e =Collegiality 

e Opportunities for professional growth 
e Supervisor support 

e = Clarity 

e Reward system 

e Decision-making influence 

e Goal consensus 


e =©Task orientation 


7 The time frame for the CCL Project permits only short-term outcomes of the BSC-SEL to be assessed within this initial feasibility 
study. Future studies of the BSC-SEL model may permit assessment of long-term outcomes and impact. 
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e = Physical environment 


e = Innovativeness 


Time 1 and Time 2 Survey 


As part of the feasibility study data collection, the Feasibility Study Team developed and fielded asurvey 
with staff at the ECE centers that participated in the CCL project (both Core BSC Team members and 
additional center staff). The purpose of the survey was to answer key implementation and outcomes 
research questions that could not be answered through a secondary measures review nor observation only. 
We included some standardized, proprietary measures into asingle survey so that from astaff person’s 
perspective, they were only completing one survey. The list of the separate measures included in the full 
survey includes: 


e Background Information. All respondents to the survey were asked about education level (highest level 
of education attained and the degree field), age, sex, race and ethnicity, and income (total household 
income level). Some of these questions overlap with the ECWES but were administered again in the 
Time 1 and Time 2 surveys so that individual responses could be captured by the Feasibility Study Team 
rather than aggregated at the center-level. 


e Early Childhood Job Satisfaction Survey (ECJSS; Bloom, 2010). The Feasibility Study Team used the Early 
Childhood Job Satisfaction Survey to measure the discrepancy between existing and ideal working 
conditions as perceived by the employee. The tool measures five unique facets of the working 
environment. Respondents were asked to rate 50 items using a Likert scale about the extent to which 
they agreed or disagreed with different aspects of their job. Example items included “I feel encouraged 
and supported by my colleagues” and “My supervisor respects my work.” In part II, respondents rated 
five aspects of their position using a Likert scale as being ideal or not ideal. Example items included 
“Relationship with co-workers” and “Relationship with supervisor.” In part III, respondents were asked 
to check three job characteristics they value most for personal and professional fulfillment. Examples 
included “Colleagues-working with people | like,” and “Security or the assurance that my position is 
secure.” In part IV, respondents were asked to write their top two satisfactions and frustrations with 
their current job. Lastly, in part V, respondents answered questions regarding their commitment to their 
center and the early childhood field. 


e Psychological Safety (Edmondson, 1999). The Psychological Safety measure examines staff's feelings of 
psychological safety and learning behavior in work teams. The survey measures the trust staff have in 
each other not to gain personal advantage at someone else’s expense. Example items included, “If you 
makea mistake at this center, it is often held against you,” and “Teachers at this center feel it is safe to 
takea risk (e.g. trying something new in the classroom).” 


e Maslach Burnout Inventory Educators Survey - Emotional Exhaustion subscale (Maslach et al., 1997). The 
Feasibility Study Team used the Emotional Exhaustion subscale of the Maslach Burnout Inventory 
(MBI), to assess how staff members felt about their job and their reactions to work. The rationale for 
including burnout was that it may be relevant for successful participation in the BSC. Participants were 
asked to rate how often items like, “I feel emotionally drained from my work” and “Working with people 
all day is really astrain for me,” were true for them. The Maslach Burnout Inventory has three subscales 
in all: Emotional Exhaustion, Personal Accomplishments, and Depersonalization. To decrease burden on 
participants, the Feasibility Study Team decided to only use the Emotional Exhaustion subscale, as it has 
the strongest psychometric properties. 


e Perceived Problems (Johnston, 1984, as adapted by Bloom). The Perceived Problems questionnaire is a 
checklist about teachers’ perceptions of day-to-day problems preschool teachers experience. We used 
selected items from this scale to identify teachers’ areas of concern in social and emotional behavior in 
their classrooms. The full questionnaire includes 45 yes/no items; however, for the purpose of the 
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feasibility study, we administered only 10 items that pertained to social and emotional learning (SEL) 
and development. Sample items included, “I have a problem getting children to do what | ask them to 
do,” and “I havea problem controlling the noise or energy level in the room.” 


e Beliefs Around Social and Emotional Learning (Brackett et al., 2012, adapted). The purpose of this survey 
was to gather background information about teachers, including experiences with professional 
development and beliefs about SEL that may be useful for understanding levels of participation in the 
BSC. Everyone except parents on Core BSC teams were asked questions about professional 
development, including the topic area of trainings attended in the past two years and the number of 
continuing education credits received within the past two years. A section in the questionnaire also 
asked about quality improvement initiatives the respondent had engaged in within the past two years 
and the respondent’s perception about how useful they were. Finally, this survey captured beliefs about 
SEL using items adapted from Brackett et al. (2012). 


e Teachers’ Sense of Efficacy Scale (Tschannen-Moran & Woolfolk Hoy, 2001; VandeWiele, 2001, adapted). 
The Feasibility Study Team used an adaptation of the Teachers’ Sense of Efficacy Scale to assess self- 
efficacy in teachers. The scale was adapted for early childhood teachers and was used to better 
understand factors relating to successful participation in the BSC. The question stem and response 
categories were drawn from Tschannen-Moran and Woolfolk Hoy (2001) andthe introductory wording 
and many of the individual questions were drawn from VandeWiele (2001). Sample items included, 
“How much can you do to make positive changes in your care setting?” and “How much can you do to 
calm achild who is upset?” Only teachers were asked to complete this section. 


e Directors’ Efficacy Scale from the Directors’ Role Perception Survey (Bloom, 2019). The Feasibility Study 
Team used sections from the Directors’ Role Perception Survey to address self-efficacy in directors. 
Only subsections from the Directors’ Role Perception Survey were used to better understand factors 
relating to successful participation in the BSC. Topics addressed in this scale focused on leadership and 
management, including managing staff, using data to promote continuous quality improvement, and 
implementing developmentally-appropriate curriculum. Only directors completed this section. 


e — Self-report of BSC Activities. In the Time 2 survey only, we askedall participants to self-report on their 
participation in BSC elements (e.g., attending a Learning Session, receiving coaching from a BSC 
Implementation Team member or faculty, doing a PDSA, etc.). The information was used by the 
Feasibility Study Team to determine 1) the extent of BSC participation (among Core BSC Team 
members) and 2) the spread of BSC practices (among others within the participating centers who were 
not themselves participants in the Core BSC Team). Additionally, this section of the survey asked 
participants about the experience of their BSC team’s participation in the BSC-SEL (e.g., engaging in 
Learning Session interactions, soliciting ideas and feedback, using PDSAs, etc.). 


e = Intra-organizational and inter-organizational learning activity items (Nembhard, 2012). The team learning 
climate was captured using Nembhard’s intra-organizational and inter-organizational learning activity 
items. The intent was to capture the team learning climate, both within teams and between teams. Intra- 
organizational items askedrespondents to indicate the extent to which their team used PDSAs and 
solicited and exchanged ideas and feedback with others at their center. Inter-organizational items asked 
respondents to indicate the extent to which their teams interacted with others at Learning Sessions and 
on the online discussion forum (i.e., Basecamp), used monthly metrics, and joined affinity group calls. 


Interviews and focus groups with Key Informants 


Key informant interviews and focus groups with directors, Core BSC Team members, Implementation Team 
members, and faculty provided additional qualitative information on the facilitators and barriers to 
implementing the BSC-SEL. 
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Key Informant Interviews with Center Directors 


The Feasibility Study Team developed an in-depth interview protocol that was conducted with center 
directors (or an alternative representative for each center) at two time points (i.e., midway through the BSC 
process and shortly after the final Learning Session). The purpose of the in-depth interview was to address 
topics that could not be answered by survey or observation. In-depth interviews were scheduled separately 
with each director and took place over the phone. Topics covered in the key informant interviews with 
center directors included, but were not limited to: 


e perceptions of organizational readiness for the BSC asa quality improvement methodology 
e organizational culture and spread of SEL practices within the center 

e _ the feasibility of implementing aBSC at their center 

e the cost of participating in the BSC 

e perceptions and experiences with “spread” 


e directors’ perceptions of sustainability of BSC and SEL practices beyond the fourth Learning Session 


Key Informant Interviews with Implementation Team 


The Feasibility Study Team developed a semi-structured interview protocol that was used with members of 
the Implementation Team in group and one-on-one interviews (depending on availability) and administered 
at two time points (i.e., midway through the BSC process and shortly after the final Learning Session). The 
purpose of the interviews was to gather the Implementation Team’s perceptions of working with the 
individual Core BSC teams and to document lessons learned about the BSC in ECE settings. Topics included 
but were not limited to: 


e = their definition of successful participation in the BSC 

e perceptions of readiness of the Core BSC teams and how teams established their goals 
e successes and challenges 

e strengths and limitations of a 12-month timeline for the BSC 

e model adaptations 

e resources and costs 


e thoughts about sustainability in these centers, as well as expansion beyond the study centers 


Key Informant Interviews with Faculty 


The Feasibility Study Team developed a semi-structured interview protocol that was conducted with 
members of the BSC-SEL faculty in group and one-on-one interviews (depending on availability) and 
administered midway through the BSC process and again after the final Learning Session. The purpose of 
the interview was to gather the faculty’s perceptions of working with the individual Core BSC teams and to 
document lessons learned about supporting SEL practices in ECE settings. Topics included but were not be 
limited to: 


e their definition of successful changes to SEL practices 
e perceptions of readiness of the Core BSC teams 


e successes and challenges 
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e strengths and limitations of a 12-month timeline for making changes to SEL practices 


e thoughts about sustainability of SEL practices in these centers 
Core BSC Team Focus Group 


The Feasibility Study Team developed a focus group protocol that was conducted with staff at the ECE 
centers who participated in the Core BSC teams and was administered after the final Learning Session. The 
purpose of the focus group was to answer key implementation and outcomes research questions that could 
not be answered through a secondary measures review nor observation only. Topics covered by the focus 
group protocol included, but were not limited to: 


e BSC elements that were beneficial and challenging for BSC Team members 
e strengths and limitations of a 12-month timeline for the BSC 


e perceptions and experiences with “spread” 


Observations”? 


Observations were conducted before and after the implementation of the BSC-SEL and provided an 
objective assessment of changes in SEL practices in center classrooms. 


Observations were conducted to capture and describe teaching practices related to SEL at two time points 
in the BSC-SEL process (i.e., soon after the first Learning Session and againafter the completion of the final 
Learning Session). The observational tools selected for the feasibility study were the Teaching Pyramid 
Observation Tool (TPOT) and the Pyramid Infant-Toddler Observational Scale (TPITOS) for preschool and 
infant/toddler classroom observations, respectively, as these observational measures are most closely 
aligned with the Center for the Social Emotional Foundations for Early Learning Pyramid Model on which 
the BSC-SEL Collaborative Change Framework and Driver Diagram were based. Two trained observers 
conducted observations in preschool and infant-toddler classrooms of Core BSC Team teachers and 
preschool and infant-toddler classrooms of other teachers at participating centers. This allowed for 
comparisons across classrooms of Core BSC Team teachers and others at participating centers. 
Observations in classrooms of other teachers at the centers were also used to determine “spread” in quality 
improvement practices beyond Core BSC teams. 


Teaching Pyramid Observation Tool (TPOT; Hemmeter et al., 2013) 


The TPOT was used as the observational tool in a selection of preschool classrooms (with children between 
the ages of two and five years old) at participating centers. It has three subscales with items documenting 
how preschool teachers implement the Pyramid Model. The first subscale is called Key Practices and 
includes items such as, “Teachers engage in supportive conversations with children” and “Teacher 
acknowledges the children’s communication tohim or her.” The second subscale is Red Flags. An example 
Red Flag that could be observed is, “Transitions are more chaotic than not.” The third subscale is Responses 
to Challenging Behavior. It includes items such as, “Teacher responds to children by stating the expected 


99 This report does not include recruitment methodology nor findings from the observations. The TPOT and TPITOS were designed for 
use in both research and quality improvement efforts. However, these tools are primarily for professional development purposes, 
where a coach discusses the results withthe teacher. While it was feasible to conduct on-site observations in the center classrooms 
that participated in the BSC-SEL, the data collected from the observations were difficult tointerpret in the context of evaluating 
outcomes. See more about conducting on-site observations to measure teacher-level outcomes about sodal and emotional teaching 
practices in the Lessons Learned about Feasibility section. 
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behavior in positive terms (i.e., what to do) or providing instruction in an acceptable behavior.” The tool 
relies both on observed behaviors and teacher report to determine item scores. The observation takes 
approximately two hours to fully complete. At the end of the observation, observers conducted a 15- to 20- 
minute interview with the lead teacher to collect additional information and determine final scores. If the 
teacher was unavailable to complete the interview immediately after the observation, the observer 
arranged tocomplete the interview over the phone as soon as possible. 


The Pyramid Infant-Toddler Observation Scale (TPITOS; Bigelow et al., 2019) 


The TPITOS was used as the observational tool in infant and toddler classrooms. It involves three main 
elements: 1) observing for red flags; 2) observing specific routines (i.e., free play, feeding/mealtime, 
structured group activity) and then assessing child engagement; and 3) observing specific routines (i.e., free 
play, feeding/mealtime, physical care routine, and structured group activity) and then rating behavioral and 
environmental items. Example items include “Adults provide children with opportunities to make choices 
(e.g. ‘this book or this book’ or ‘you can sit and listen to astory or play with the toys’)” and “Adults show 
physical affection toward children and smile at them.” 


Breakthrough Series Collaborative artifacts 


Outputs of the BSC-SEL, such as documentation of attendance at and content of Learning Sessions and 
affinity group meetings, or number and type of PDSAs completed by participating Core BSC Team 
members—were all collected as artifacts of BSC implementation and contributed to the assessment of 
whether and how ECE centers engaged in BSC activities and changed their structures and work processes 
over the course of their participation in the BSC-SEL. Furthermore, observations of the Learning Sessions 
provided insights into the relational dynamics within and across organizations, and how those may change 
over the course of an organization’s participation in a BSC. 


The following BSC artifacts contributed to the Feasibility Study Team’s understanding of level of 
participation in the BSC: 


e Attendance records at BSC activities noted in the Implementation Tracker 
e Monthly Metrics Spreadsheets 


e Analysis of discussion board postings 
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Appendix D. Collaborative Change Framework 


A Breakthrough Series Collaborative to Support Social and Emotional 
Learning in Early Education and Head Start: Collaborative Change 
Framework 


Introduction 


The Breakthrough Series Collaborative (BSC) to Support Social and Emotional Learning (SEL) in Early 
Education and Head Start was a unique quality improvement collaborative that took place over a twelve-to- 
eighteen-month period. During the collaborative, centers made improvements in their practices, policies, 
and environments to support children’s SEL, using the Pyramid Model from the Center for the Social and 
Emotional Foundations for Early Learning. The Pyramid Model has been extensively studied in early 
childhood and early childhood special education settings and is used widely in child care and Head Start 
centers. 


What is a Breakthrough Series Collaborative? 


The Breakthrough Series Collaborative methodology differs from other ECE quality improvement initiatives 
as it supports change by building capacity and recognizing leaders across an organization, rather than 
through a single coach or consultant. It promotes change through collaborative teams that include staff at all 
levels and families and engages teams in problem-solving that takes local context into account. A goal of the 
BSC is to spread and sustain change. BSCs have supported practice and process improvements in the health, 
child trauma, and child welfare fields, but has not yet been widely testedin ECE. It includes: 


e Learning opportunities toenhance knowledge and skills in SEL and quality improvement 


e Collaborative teams (teachers, administrators, and families) working with faculty and BSC staff to 
become skilled at quality improvement, to set specific goals and test improvements 


e AChange Framework that is the roadmap for teams to move beyond concepts in SEL to actual change 
strategies 


e Asupportive learning collaborative where teams network, learn from one another, share ideas and 
problem solve challenges together 


e Collecting and using data to learn about what is going well and where improvements are needed 


e Small tests of change, or Plan-Do-Study-Act cycles (PDSAs), to build evidence to make and spread 
improvements in early childhood care and education settings 


What is the Change Framework? 


The Change Framework is the content anchor for all work that is done by teams in the BSC. It serves a 
roadmap throughout the BSC to guide teams’ assessment of their strengths and challenges, where they can 
improve, and what changes to test as well as tracking their progress over time. The Change Framework 
describes the need and opportunity for improving SELin child care and Head Start and the overarching 
aims that the centers in the Collaborative seek to achieve through adopting evidence-based practices in 
SEL. The Driver Diagram describes what elements (primary drivers) must change ina child care or Head 
Start center and how those elements must change (secondary drivers) to achieve the aims of the BSC. This 
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Change Framework was developed by examining the research and practice guidance from the Pyramid 
Model, and gathering the expertise of faculty, coaches, and practitioners in the field. 


The need and opportunity for improving social and emotional learning practices 


From infancy onward, children’s growing ability to express, understand, and regulate emotions and to form 
social connections are essential to their development. Socially- and emotionally-competent children 
understand their own and others’ emotions, form positive relationships, and have developmentally 
appropriate levels of self-regulation. They are more likely to take advantage of learning opportunities in the 
classroom and to achieve academic success as they move through school. Social and emotional development 
has received considerable attention from the early childhood research and practice communities (e.g., 
Denham, 2006; Barrett et al., 2016; Hyson, 2004). This attention has focused on two of the overarching aims 
for teacher practices: 1) to increase positive social behavior in young children and 2) to reduce challenging 
behaviors (Hemmeter & Fox, 2008). 


It is widely recognized that SEL is an area in which instructional quality can be improved and teachers often 
express interest in additional supports (Cimino et al., 2007; Fox et al., 2003; Hemmeter & Fox, 2008). For 
example, challenging behaviors is an area for which early educators frequently seek additional professional 
development supports (Cimino etal., 2007). Early educators play a key role when it comes to supporting the 
development of young children’s social skills and positive behaviors. Early learning policies reflect a growing 
attention to enhancing teaching quality in this domain, as demonstrated in the prioritization of social- 
emotional health in the reauthorized CCDBG and in Head Start. In addition, recent policy statements from 
both Head Start» and federal agencies address expulsion and suspension practices in early care and 
education (ECE) settings (U.S. Department of Health and Human Services & U.S. Department of Education, 
2016) resulting ina growing need for teachers to be equipped with the right skills to manage challenging 
behavior in the classroom with a culturally competent approach. 


A commonly used approach to promoting SELis the Pyramid Model intervention (Fox et al., 2003). The 
Pyramid Model (Fox et al., 2003) has been extensively studied in early childhood and early childhood special 
education settings, and is used widely in child care and Head Start centers in the geographic region where 
the feasibility study took place. The model was developed to provide early educators with guidance on how 
to support young children’s social-emotional development and address challenging behavior. It clearly 
articulates the teaching practices that support children at three levels: universal/primary strategies, 
targeted/secondary strategies, and individualized/tertiary strategies. 


Overarching aims 


Over the course of the BSC, all teams will be striving to achieve three overarching aims: 


1. Increase the number of classrooms that show improvement in providing high-quality, SEL environments 
2. Increase children's social and emotional competence 
3. Reduce challenging behaviors 


>b See http://eclkc.ohs.acf.hhs.gov/policy/im/acf-im-hs-16-01 for links tonew requirements in the Head Start Performance Standards 
about expulsions and suspensions. 
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The Driver Diagram 


The five primary drivers in this Change Framework, along with their related secondary drivers, describe the 
structures, processes, practices, and norms required for SEL practices to be adopted by participating 
centers in the BSC and for teams to achieve the overarching aims. 


1. Nurturing, responsive and supportive relationships and environments are critical factors in supporting 
social and emotional development and promoting appropriate, pro-social behaviors. Children learn and 
develop through their relationships with adults. By building nurturing, positive relationships with 
children, teachers can support children’s self-esteem, sense of safety, and positively influence behavior. 
The design of classrooms, schedules and activities can contribute to promoting children’s positive 
engagement in their environments and prevent challenging behaviors. 


2. Teaching social skills and emotional competencies in the context of children's culture and language: Children 
develop social skills and emotional competencies through explicit teaching of emotional literacy, 
problem-solving skills, and positive peer relationship skills and practicing those skills with the support of 
teachers in daily interactions and when challenges arise. 


3. Family partnership to promote the adoption of evidence-based SEL practices: Families are children’s first 
teachers. It is critical for early education centers to collaborate with families to promote their children’s 
social and emotional development at home and in the center through developing positive relationships 
with parents, promoting parents’ confidence and competence, and connecting to concrete supports 
when needed. 


4. Racial equity and culturalresponsiveness in promoting the adoption of evidence-based SEL practices: To 
implement the Pyramid Model effectively, centers must address implicit bias and promote cultural 
responsiveness and equity. These are essential to nurturing positive adult-child relationships; 
supporting positive adult-adult relationships (staff-staff and staff-families); promoting positive identity 
development for children; and reducing inequities in perceptions of and responses to children’s 
behaviors, including decisions about disciplinary actions. 


5. Organizational capacity and support to promote the adoption of evidence-based SEL practices: |n order to 
support the social and emotional development of children, teachers must feel valued, respected, and 
supported by their colleagues and leadership as well as have the tools, resources and training necess ary 
to effectively implement the Pyramid Model practices. 


Figure 17 provides a visual display of the inter-relationships among the five primary drivers in this BSC. 
Although we know that the primary drivers are inter-related and inter-connected, calling them out as 
distinct helps break the more complex issues into discrete and manageable “chunks.” The drivers help teams 
recognize the need for change at various “levels” of their system. 


Promoting a Culture of Continuous Learning in Early Care and Education Settings 69 


Figure 17. How the Five Primary Drivers are related and connected 


Organizational Capacity 


F-Tale cle) eo) ele) au Organizational capacity and 
support is the “container” for 
all transformational work 


Racial Justice and 
Cultural Responsiveness 


All social and emotional 
learning must occur within 
the context of racial justice 
and cultural responsiveness 


Direct service interactions at 
the individual level that are 
integrally connected and 


overlap in many ways 


Source: Implementation Team 
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Each of these primary drivers is then further broken down into more specific opportunities and strategies 
that centers can test out in practice called Secondary Drivers, described in Figure 18 below: 
Figure 18. Driver Diagram 


Primary Drivers Secondary Drivers 


Relationships 


FYave| 
Environments 


Social Skills 
FVaveR rated (ogre) 
Competencies 


Family 
Partnership 


Racial Equity 
and Cultural 
Responsiveness 


Organizational 
Capacity and 
Support 


e Design classrooms in responsive ways 

e Develop and maintain predictable schedules, routines and activities to promote smooth 
transitions 

e Design activities to promote engagement 

e Provide directions 

e Teach behavior expectations 

eTeachers develop nurturing, meaningful relationships with children and colleagues 


e Use open, consistent communication with families 

elnvite and support parent participation in the classroom and center 

e Develop postive, respectful, meaningful, culturally affirming relationships with families 

eConnect families to concrete supports intimes of need 

e Collaborate with families to support the parent-child relationship and the families' 
promotion of their child's social and emotional competencies 

eSupport/promote parent confidence/competence (parental resilience) 


e Have and use a system to collect and review data by race/ethnicty, language, gender, and 
ability 

e Provide ongoing professional development and reflection opportunties for staff on 
cultural responsiveness, implicit bias, and race equity 


eHonor and reflect diverse identities and families inthe classroom and center 
environment 


e Develop clear policies and guidelines for preventing challenging behaviors that are free 
from bias, and ensure equity and fairness 


eCreate a culture of collaboration in the classroom and center 

e Use facilitative leadership and reflective supervision 

eSupport healthy staff, stress-reduction, adult relational competencies, self-regulation 

e Use data systems to track behavior and social and emotional competence 

eEnsure access to mental health consultation to support individualized interventions for 
children with persistent challenging behaviors, disabilities, or developmental challenges 

e Maintain effective staffing patterns (ratios, hours, breaks, time off, staffing consistency, 
primary caregiver model, continuity of care model) 

eNurture a postitve workplace climate where staff feel supported and valued 

e Provide ongoing training and coaching 


Source: Implementation Team 
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